NEP Andrews Hubs in Melbourne and Sydney
To overcome the obstacles of distance and location in remote production, NEP created centralised Internet Protocol (IP) production centres in Sydney, NSW, and Melbourne, VIC, which are named in memory of NEP Australia’s late and much-loved CEO, Keith Andrews, who championed the initiative.
The Andrews Hubs are the world’s most advanced facilities of their kind, able to cover up to seven simultaneous live events from 29 connected venues around Australia. They’re also arguably the world’s most flexible: 100 per cent IP and also multi-format for any production requirement, including 4K, high dynamic range (HDR) and any framerate.
NEP Australia’s director of technology, Marc Segar explains: “The Andrews Hubs allow for a totally distributed human resource, where the team working on a project no longer have to physically be together but can now work across many locations using modern communications tools to maintain a team approach.”
NEP had been looking at centralising production for some time, and in response to a request for proposal from a key client in August 2016 they designed, constructed and deployed the hubs within a tight 14-month timetable. While an IP infrastructure was always the logical approach, when NEP began the project the SMPTE 2110 standards were still in final draft, and no manufacturer had SMPTE 2110-compliant equipment.
“The challenge of working with ‘vapour products’ was significant, but it helped shape our exact requirements and ensured all components were tailored to our needs,” Segar adds. “Some of the world’s leading manufacturers were quick to jump on board and partner us in making the hubs concept real. There is simply no way we could have done this in the timeframe in serial digital interface (SDI), nor could we have achieved it even a year earlier: the technology just wasn’t there. We also wanted to push the boundaries of anything done before, and that’s why there isn’t a single SDI cable anywhere in our transmission chain – from the camera control unit (CCU) in our IP truck to the broadcaster.”
All design work and drawings were done by NEP’s in-house Systems Integration team, from the first drafts to the final versions including all the trucks, hybrid kits and the two production hubs themselves. Ninety per cent of cabling is fibre from NEP’s central distribution rack, with the remaining cables Cat 6 for lighting and access control, the corporate network and telephone system. To connect 29 venues to the hubs, NEP engaged Telstra to build a country-wide Distributed Production Network (DPN), with a fully redundant network at 50 Gbps.
To accommodate cameras, microphones, their operators and an on-site engineer (the only crew who now need to physically move from project to project) NEP built four new IP-based production trucks and upgraded seven existing outside broadcast vehicles to IP
A comprehensive video conferencing system enables a split production environment: cameras are located at each operational venue, but all program production staff – including the director, vision switcher, audio director, graphics and relay operators – are based at an Andrews Hub control room in Sydney or Melbourne. Those cities were chosen because of the majority broadcast professionals who work on Australia’s major sporting codes reside in one or the other: Sydney is the centre for Rugby Union, Rugby League and football (soccer), with Melbourne home to the production community supporting Australian Rules football and cricket. “So long as they are connected to our network, anyone can work from anywhere,” Segar says.
The two sites are completely redundant and can share all their resources, including replay, production switchers and graphics – in fact just about anything that an operator needs to do their job.
Proof of concept
The Andrews Hubs passed a major milestone on 21 December, 2017 in producing the world’s first live-to-air SMPTE 2110 uncompressed high definition (HD) remote production. A Football Federation Australia A-League match between Brisbane and Perth held at Brisbane’s Suncorp Stadium was produced and directed 920 kms (570 miles) away at NEP’s Sydney Hub in Eveleigh, just south of the city’s central business district.
Live uncompressed HD signals from 10 Sony HDC 4300 4K/HD cameras at the venue – including high frame rate cameras – were sent to Sydney via the Telstra DPN.
HD video conferencing supported the split production, with the entire production team aside from camera and microphone operators based at the Sydney Andrews Hub. The final program was then delivered to Fox Sports via IP for live-to-air broadcast – proving the hubs were fit for purpose and putting NEP squarely on track for the facilities to come fully online in the first quarter of 2018.
On March 10 NEP again made history as the Sydney Andrews Hub produced its first trans-continental live uncompressed broadcast: the FFA A-League match between Perth Glory and the Central Coast Mariners held at NIB Stadium in Perth: 4,000 km (2400 miles) apart.
“This was significant, as it was the first time that SMPTE 2110 had been used across that distance with a latency of just 48ms round trip: just over one frame of video,” says Segar. “It also reinforced that the hubs integrate seamlessly in a ‘regular’ weekend in which we deliver multiple events around the country from different sporting codes.”
Based in Perth was NEP’s IP 21, a 13-metre all-IP production truck, housing the only onsite staff: camera operators, audio technicians and commentators; the remaining production team members were in Sydney. Uncompressed video and audio were delivered via the Telstra DPN to produce the show, which was then sent over IP to Fox Sports for live broadcast.
NEP’s new centralised model and associated workflows have cut in half the number of crew needed to move between a typical project. The benefits also go well beyond travel cost savings. For example, the team producing the Perth A-League match on Saturday night were able to do another job the following day, which otherwise would have been lost to travel. It also means fewer days spent away from home, more time for training, and the chance for crew to work on a wider variety of projects.
In a roundabout way, the hubs offer an environment akin to that of a traditional broadcaster, where staff come to the facility each day. “We’ve taken care to create a comfortable, practical environment that people want to work in,” says Segar. “Staff like the size and relative quiet in the control room. It makes for a more relaxed, happier production team, and we make better TV because of it.”
In the year from March 2018, NEP Australia’s Andrews Hubs are set to produce more than 250 events, including the FIFA World Cup from Russia, with 500 or more events in 2019.
The company is also looking to deploy its hub capabilities beyond Australia’s shores, and in late April delivered the world’s first remote production across the Pacific. The successful trial was conducted over four days between the Sydney Andrews Hub and Telstra’s Los Angeles datacentre – more than more than 7500 miles (12,000+ km) apart – using VC-2 ultra-low-latency compression technology and the Telstra DPN.
30 HD camera feeds in LA were linked via diverse and hitless 10 Gbps circuits on the DPN, with the production taking place in Sydney, overcoming what until now was considered an insurmountable distance. The tests confirm that the Andrews Hub control rooms in Sydney can efficiently produce broadcast events held in Los Angeles and, arguably, beyond.
Segar notes: “Our crew at the Sydney Andrews Hub perceived no appreciable difference between covering the trial event in LA and the remote broadcast of the live A-League match from Perth last month. This trial proves anyone can work from anywhere while connected to our network, whether in Australia or on the other side of the planet.”
Telstra’s Head of Broadcast Services, Trevor Boal, adds that as Telstra’s network capacity and reach continues to grow, “it’s conceivable we could traverse global distances uncompressed in just a few years.”
The successful trans-Pacific trial is just the start of NEP’s extensive program to trial other locations around the world, including Japan and the UK.