Distributed production is no longer remote!
Posted on Aug 4, 2022 by Samara Husbands
In this Xtreme Round Table, we discuss a new approach to production, set to revolutionise sports broadcast
Neal Romanek: To start, how exactly would you define ‘remote production’?
Norbert Paquet: There is another term I employ: ‘distributed production’. That gives a better idea of how we’re distributing resources. In terms of these, two main elements compose a production process. The first is technical resources – acquiring, processing or distributing content. Then, there are people who operate or manage the technical infrastructure.
Claire Wilkie: That’s right. It is anything that separates your MCR or gallery from the studio, so they are operating in different locations. It streamlines your workflow, plus it’s sustainable.
Tim Puschkeit: In the early days, we called it a remote production. However, you perhaps only used a few services remotely, while everything else was on-site with OB vans. It changed, especially throughout the pandemic. That was our comfort zone and the established business practice over the past couple of years. But we were asked to leave our comfort zone and do it in a different, new way.
Neal Romanek: What are the real benefits to doing production in this distributed way?
Robert Erickson: It comes down to efficiency and duty cycle. With college football or baseball, you might have 70 or 80 games a day. Networks, such as ESPN, traditionally fly a crew to every single production. You want to do a baseball game: you get a truck, you fly in a crew. You want to do another baseball game: you get another truck, you fly in another crew. They are there for a two-hour production, then go back to their hotel room, before flying home the next day.
But remote production introduces efficiency. What if I can build a hub in LA, Charlotte or London? I can assemble a large staff with a technical director, audio engineer, graphics operator, producer and assistant director. All those people can do one show. They can take a break and walk back into that same studio an hour later for another show. You can do back-to-back shows, and that capital asset you invested millions of dollars in can have an 80% usage cycle.
Claire Wilkie: I’m not an engineer. I’m a girl with a vision. I find remote production really exciting. I love live production and coming up with challenging, mad ideas. Remote production opens up so much freedom for that.
We use a lot of Blackmagic Design kit that proves scalable and modular. On top of that, we don’t have a legacy of big broadcast trucks – we are not one of those behemoth companies. With remote production, you really are unlimited in what you can do now. With the tech becoming smaller, it means a more level playing field, and this is something that’s necessary.
Tim Puschkeit: Even though I’ve been working in this business a while, sometimes I still think it’s unreal that you can operate cameras located in Sydney or Paris from London or Germany. But it’s our bread and butter now – and it’s the future.
Remote production does give you more challenges, requiring clearer communication. The camera operator may be sitting in a different time zone, speaking a different language.
Norbert Paquet: Distributed production is modular. But considering exactly where you place the different building blocks of the production process is important. That is the acquisition, production and processing of the content, as well as distribution, plus the people required.
A director might appreciate working close to his home if the stadium is far away. But others prefer to go the stadium to see the atmosphere and talk to camera operators face to face. There is that flexibility with remote production; you can put the right people in the right places, based on the production workflow you want to achieve.
Neal Romanek: How might remote production actually affect how – and where – people work?
Robert Erickson: There’s been technology for five or ten years that has allowed us to do remote productions efficiently. But the way we’ve been doing production for eternity has worked – and broadcasters were reticent to make the change. You’ve always had engineers that want improvements, but someone at the network was likely to say: “I get that, but this is how we’ve done things. This is a premier event and we’re not going to risk messing up our cash cow.”
Covid-19 opened up the door. Engineers could say: “Let’s try it this way. We’ve never done it, but the risk is actually pretty low.” It introduced the idea of innovation to senior management, raising the industry’s tolerance for failure – it’s never been a big fan of failure. But true innovation comes with the idea that you learn from possible failure, then you move forward.
Tim Puschkeit: Big events, like the Super Bowl, Olympics or World Cup – where you have a production of four weeks – will remain on-site. Maybe you can share synergies, using established remote production units as well, but a lot of things will still happen on-site. For the events that come back on a weekly, monthly or annual period, with a more or less standardised set-up, I definitely see remote production as the future.
Steven Dargham: It was always something broadcasters wanted. We’d come in and do workshops and trials. They would discuss it, but it never got to the working stage. Now, it’s a necessity. Last year’s Tokyo Olympics were completely different. Every single broadcaster we were involved with did everything from back at home, with nothing at the International Broadcast Centre. They’re changing the workflow. Keep in mind, we couldn’t do that without an advancement in the underlying technology and telco infrastructure.
In this Xtreme Round Table, we discuss a new approach to production, set to revolutionise sports broadcast
Neal Romanek: To start, how exactly would you define ‘remote production’?
Norbert Paquet: There is another term I employ: ‘distributed production’. That gives a better idea of how we’re distributing resources. In terms of these, two main elements compose a production process. The first is technical resources – acquiring, processing or distributing content. Then, there are people who operate or manage the technical infrastructure.
Claire Wilkie: That’s right. It is anything that separates your MCR or gallery from the studio, so they are operating in different locations. It streamlines your workflow, plus it’s sustainable.
Tim Puschkeit: In the early days, we called it a remote production. However, you perhaps only used a few services remotely, while everything else was on-site with OB vans. It changed, especially throughout the pandemic. That was our comfort zone and the established business practice over the past couple of years. But we were asked to leave our comfort zone and do it in a different, new way.
Neal Romanek: What are the real benefits to doing production in this distributed way?
Robert Erickson: It comes down to efficiency and duty cycle. With college football or baseball, you might have 70 or 80 games a day. Networks, such as ESPN, traditionally fly a crew to every single production. You want to do a baseball game: you get a truck, you fly in a crew. You want to do another baseball game: you get another truck, you fly in another crew. They are there for a two-hour production, then go back to their hotel room, before flying home the next day.
But remote production introduces efficiency. What if I can build a hub in LA, Charlotte or London? I can assemble a large staff with a technical director, audio engineer, graphics operator, producer and assistant director. All those people can do one show. They can take a break and walk back into that same studio an hour later for another show. You can do back-to-back shows, and that capital asset you invested millions of dollars in can have an 80% usage cycle.
Claire Wilkie: I’m not an engineer. I’m a girl with a vision. I find remote production really exciting. I love live production and coming up with challenging, mad ideas. Remote production opens up so much freedom for that.
We use a lot of Blackmagic Design kit that proves scalable and modular. On top of that, we don’t have a legacy of big broadcast trucks – we are not one of those behemoth companies. With remote production, you really are unlimited in what you can do now. With the tech becoming smaller, it means a more level playing field, and this is something that’s necessary.
Tim Puschkeit: Even though I’ve been working in this business a while, sometimes I still think it’s unreal that you can operate cameras located in Sydney or Paris from London or Germany. But it’s our bread and butter now – and it’s the future.
Remote production does give you more challenges, requiring clearer communication. The camera operator may be sitting in a different time zone, speaking a different language.
Norbert Paquet: Distributed production is modular. But considering exactly where you place the different building blocks of the production process is important. That is the acquisition, production and processing of the content, as well as distribution, plus the people required.
A director might appreciate working close to his home if the stadium is far away. But others prefer to go the stadium to see the atmosphere and talk to camera operators face to face. There is that flexibility with remote production; you can put the right people in the right places, based on the production workflow you want to achieve.
Neal Romanek: How might remote production actually affect how – and where – people work?
Robert Erickson: There’s been technology for five or ten years that has allowed us to do remote productions efficiently. But the way we’ve been doing production for eternity has worked – and broadcasters were reticent to make the change. You’ve always had engineers that want improvements, but someone at the network was likely to say: “I get that, but this is how we’ve done things. This is a premier event and we’re not going to risk messing up our cash cow.”
Covid-19 opened up the door. Engineers could say: “Let’s try it this way. We’ve never done it, but the risk is actually pretty low.” It introduced the idea of innovation to senior management, raising the industry’s tolerance for failure – it’s never been a big fan of failure. But true innovation comes with the idea that you learn from possible failure, then you move forward.
Tim Puschkeit: Big events, like the Super Bowl, Olympics or World Cup – where you have a production of four weeks – will remain on-site. Maybe you can share synergies, using established remote production units as well, but a lot of things will still happen on-site. For the events that come back on a weekly, monthly or annual period, with a more or less standardised set-up, I definitely see remote production as the future.
Steven Dargham: It was always something broadcasters wanted. We’d come in and do workshops and trials. They would discuss it, but it never got to the working stage. Now, it’s a necessity. Last year’s Tokyo Olympics were completely different. Every single broadcaster we were involved with did everything from back at home, with nothing at the International Broadcast Centre. They’re changing the workflow. Keep in mind, we couldn’t do that without an advancement in the underlying technology and telco infrastructure.
Neal Romanek: What has been some of your practical remote production experience recently?
Steven Dargham: For the America’s Cup last year, we had two of the events in Europe cancelled due to Covid-19. We ended up doing the America’s Cup without a single person from Telstra on-site.
Tim Puschkeit: Riedel worked on the America’s Cup project, too. We started four years ago. The plan was to do a series in six or seven different venues across Europe and the US. We even did recce missions and site visits. Instead, we spent six months in Auckland, New Zealand. But, in that time, we did prototype engineering. It was challenging, because we follow a 360-degree approach. We not only did the broadcast, but the technology on the boats and the race management system. We already had some remote solutions installed, with remote engineering and data logging on the boats, plus the cloud was a remote solution. We also had remote edit suites operating in Europe that could create news feeds overnight.
Nonetheless, due to restrictions, not a single broadcaster was able to go to the event last year. We used Telstra’s redundant fibre to feed out to local broadcasters. Steven and I spent a lot of time on the phone between Auckland and where he was in Sydney, trying to figure things out. In the end, we used everything we had, because the local broadcasters asked for so many things: individual signals, a different voiceover and other unilateral services. We were flat out. That’s something we need to learn. Infrastructures and set-up need to be prepared for big events in the future.
Norbert Paquet: One thing has been the production planning and coordination of resource. Usually, the main conversation is around latency management: how do we manage the different latencies introduced by remote or distributed operation?
Communication is fundamental to everything. As soon as you introduce latency into communication, it impacts the entire production and value chain. We’ve looked at solutions that manage the overall set of latencies introduced by the different paths the audio takes, and then how you resynchronise that in the end.
We had discussions with customers around codecs and compression ratios, because bandwidth is also an element to consider. Some people have one gig uplink – some don’t – so you must pay attention to that.
Neal Romanek: How can remote production help smaller companies or niche broadcasters?
Robert Erickson: Here’s an amazing example. There is this ridiculous drinking game in North America called cornhole, where people throw sandbags at a wooden platform with a hole in it. During the first lockdowns, cornhole got big ratings numbers throughout North America. The part I thought was really cool was that it was a completely automated production, with AI literally choosing the camera shots, and an audio guy at a board riding the levels.
The fact they were using remote production and some cool new tools allowed them to take something as mental as cornhole, and make money from it. And people watched it! That’s the incredible part – granted they started watching it less once football and other sports started coming back. It was a really cool use of going down scale and trying new technologies.
Steven Dargham: Remote production allows tier-two sports to do more for less. Producing a small event, with one or two cameras controlled using artificial intelligence and remote production, you can actually put a high-quality show on air. The surprises CEOs and CTOs are getting is that they don’t necessarily have to pay more. That’s where the difference comes. It’s a game-changer – not just for major events, but tier-two ones as well. You don’t require people to manage cameras. You can control a football match these days using artificial intelligence, without anyone driving to the stadium.
Claire Wilkie: It’s exciting to talk about doing production in circumstances we can’t control – for long-distance cycle racing, motor sport, ultramarathons, not just your top-tier sports. In 2019, UCI para-cycling came to Yorkshire in the UK. For the first time, thanks to remote production, it was streamed and broadcast live, using 4G networks.
My team ran the remote production. We covered six races in one day, before the rest of the UCI cycling coverage. That is what really excites me: offering exposure to sports that usually can’t afford that level of coverage and storytelling.
Neal Romanek: What has been some of your practical remote production experience recently?
Steven Dargham: For the America’s Cup last year, we had two of the events in Europe cancelled due to Covid-19. We ended up doing the America’s Cup without a single person from Telstra on-site.
Tim Puschkeit: Riedel worked on the America’s Cup project, too. We started four years ago. The plan was to do a series in six or seven different venues across Europe and the US. We even did recce missions and site visits. Instead, we spent six months in Auckland, New Zealand. But, in that time, we did prototype engineering. It was challenging, because we follow a 360-degree approach. We not only did the broadcast, but the technology on the boats and the race management system. We already had some remote solutions installed, with remote engineering and data logging on the boats, plus the cloud was a remote solution. We also had remote edit suites operating in Europe that could create news feeds overnight.
Nonetheless, due to restrictions, not a single broadcaster was able to go to the event last year. We used Telstra’s redundant fibre to feed out to local broadcasters. Steven and I spent a lot of time on the phone between Auckland and where he was in Sydney, trying to figure things out. In the end, we used everything we had, because the local broadcasters asked for so many things: individual signals, a different voiceover and other unilateral services. We were flat out. That’s something we need to learn. Infrastructures and set-up need to be prepared for big events in the future.
Norbert Paquet: One thing has been the production planning and coordination of resource. Usually, the main conversation is around latency management: how do we manage the different latencies introduced by remote or distributed operation?
Communication is fundamental to everything. As soon as you introduce latency into communication, it impacts the entire production and value chain. We’ve looked at solutions that manage the overall set of latencies introduced by the different paths the audio takes, and then how you resynchronise that in the end.
We had discussions with customers around codecs and compression ratios, because bandwidth is also an element to consider. Some people have one gig uplink – some don’t – so you must pay attention to that.
Neal Romanek: How can remote production help smaller companies or niche broadcasters?
Robert Erickson: Here’s an amazing example. There is this ridiculous drinking game in North America called cornhole, where people throw sandbags at a wooden platform with a hole in it. During the first lockdowns, cornhole got big ratings numbers throughout North America. The part I thought was really cool was that it was a completely automated production, with AI literally choosing the camera shots, and an audio guy at a board riding the levels.
The fact they were using remote production and some cool new tools allowed them to take something as mental as cornhole, and make money from it. And people watched it! That’s the incredible part – granted they started watching it less once football and other sports started coming back. It was a really cool use of going down scale and trying new technologies.
Steven Dargham: Remote production allows tier-two sports to do more for less. Producing a small event, with one or two cameras controlled using artificial intelligence and remote production, you can actually put a high-quality show on air. The surprises CEOs and CTOs are getting is that they don’t necessarily have to pay more. That’s where the difference comes. It’s a game-changer – not just for major events, but tier-two ones as well. You don’t require people to manage cameras. You can control a football match these days using artificial intelligence, without anyone driving to the stadium.
Claire Wilkie: It’s exciting to talk about doing production in circumstances we can’t control – for long-distance cycle racing, motor sport, ultramarathons, not just your top-tier sports. In 2019, UCI para-cycling came to Yorkshire in the UK. For the first time, thanks to remote production, it was streamed and broadcast live, using 4G networks.
My team ran the remote production. We covered six races in one day, before the rest of the UCI cycling coverage. That is what really excites me: offering exposure to sports that usually can’t afford that level of coverage and storytelling.
Neal Romanek: What will remote – or distributed – production be like in five years?
Tim Puschkeit: Remote production will always have to be a consideration in the future. It isn’t the key to success, nor the answer to all our questions. Remote stands and falls, at the end of the day, on what the customer or broadcaster want to achieve. It’s all about emotions.
We’re already starting to go back to a bit more travelling, so it will be a balance. The discussion will move to setting priorities and deciding what is best for the next project.
Robert Erickson: The biggest difference is that we’ll just call it ‘production’. Technologies we have developed today for remote production are going to be innately baked into everything we do. It’s just a toolset that we have to create the content required.
On the engineering side, there is a unique challenge. Our job is to allow creative people to make the best content they can possibly imagine. Do you think a TD really cares if his switcher is a big Grass Valley keyframe sitting in a truck, or a bunch of compute in AWS, if it gets the job done? They just want to focus on what they’re doing. We still make switchers, and have to do some things in hardware, but if you want to do a medium or small production in the cloud, we can do that. A lot of the things we still have to do in hardware today, we’ll be able to do in software.
Claire Wilkie: I ask all my clients: “How big is your imagination? What is the most outlandish project you can think of?” Then, we look at the tools to see how to make it happen.
It comes down to that translation between technology and creativity. Half of my job is being able to do that for production companies or producers. They have an idea and say: “I want to live stream from the North Pole or a mountain in Italy. I have no idea if I can do that. Is that possible?” I love that question. Of course it is, because we have unlimited technology available and it’s always being innovated.
Everything is going to get better. Delays will be smaller and the most outlandish ideas are going to happen. That’s why I’m in business.
Norbert Paquet: Cloud, remote, IP, you name it, they will all just be part of the way we’re working – a mixture of ways to create content. The methods of working and toolsets will be put together based on the objective of the creatives. There will be a lot of flexibility in what people pick and choose. That’s why I like this distributed production model. You pick your resource – whether it’s a creative brain, the engineering or processing back ends – you put them together and you go. The options palette will just get wider.
Steve Dargham: I like what Claire said. In 2012, James Cameron wanted to go to the Mariana Trench, the deepest point of the Earth. And guess what? We connected him live, from the Mariana Trench, all the way from National Geographic HQ.
Let me take you back to three years ago, when Fox Sports Australia decided on distributed production. For the past three years, the National Rugby League, Australian soccer and AFL have been done remotely from Melbourne or Sydney. In five years, there will be more AI, but remote will just be part of production.
This article first featured in issue 01 of Xtreme.
Roundtables archives
Neal Romanek: What will remote – or distributed – production be like in five years?
Tim Puschkeit: Remote production will always have to be a consideration in the future. It isn’t the key to success, nor the answer to all our questions. Remote stands and falls, at the end of the day, on what the customer or broadcaster want to achieve. It’s all about emotions.
We’re already starting to go back to a bit more travelling, so it will be a balance. The discussion will move to setting priorities and deciding what is best for the next project.
Robert Erickson: The biggest difference is that we’ll just call it ‘production’. Technologies we have developed today for remote production are going to be innately baked into everything we do. It’s just a toolset that we have to create the content required.
On the engineering side, there is a unique challenge. Our job is to allow creative people to make the best content they can possibly imagine. Do you think a TD really cares if his switcher is a big Grass Valley keyframe sitting in a truck, or a bunch of compute in AWS, if it gets the job done? They just want to focus on what they’re doing. We still make switchers, and have to do some things in hardware, but if you want to do a medium or small production in the cloud, we can do that. A lot of the things we still have to do in hardware today, we’ll be able to do in software.
Claire Wilkie: I ask all my clients: “How big is your imagination? What is the most outlandish project you can think of?” Then, we look at the tools to see how to make it happen.
It comes down to that translation between technology and creativity. Half of my job is being able to do that for production companies or producers. They have an idea and say: “I want to live stream from the North Pole or a mountain in Italy. I have no idea if I can do that. Is that possible?” I love that question. Of course it is, because we have unlimited technology available and it’s always being innovated.
Everything is going to get better. Delays will be smaller and the most outlandish ideas are going to happen. That’s why I’m in business.
Norbert Paquet: Cloud, remote, IP, you name it, they will all just be part of the way we’re working – a mixture of ways to create content. The methods of working and toolsets will be put together based on the objective of the creatives. There will be a lot of flexibility in what people pick and choose. That’s why I like this distributed production model. You pick your resource – whether it’s a creative brain, the engineering or processing back ends – you put them together and you go. The options palette will just get wider.
Steve Dargham: I like what Claire said. In 2012, James Cameron wanted to go to the Mariana Trench, the deepest point of the Earth. And guess what? We connected him live, from the Mariana Trench, all the way from National Geographic HQ.
Let me take you back to three years ago, when Fox Sports Australia decided on distributed production. For the past three years, the National Rugby League, Australian soccer and AFL have been done remotely from Melbourne or Sydney. In five years, there will be more AI, but remote will just be part of production.
This article first featured in issue 01 of Xtreme.
Roundtables archives
