1 Cooper Final Draft 64kbps
  [intro music]
Participants:
Jesse Cooper
Communications and Information Technology Project Manager 
City of Phoenix, Arizona

Ben Krauss
Public Safety Technology Specialist
SEARCH, The National Consortium for Justice Information and Statistics

Podcast length: 35:26

Ben Krauss:  The following is another conversation in our series of podcasts on lessons learned and best practices from projects funded through COPS technology grants. These podcasts are presented by SEARCH, The National Consortium for Justice Information and Statistics. This series is made possible through funding from the U.S. Department of Justice Office of Community Oriented Policing Services, COPS. Today's topic is managing communication systems lifecycles and the multivendor environment. Our guest today is Jesse Cooper, Communications and Information Technology Project Manager for the City of Phoenix. I'm Ben Krauss, a Public Safety Technology Specialist for SEARCH and moderator for this podcast.
The City of Phoenix and nearby jurisdictions have been leaders in standards-based shared radio systems for critical communications. They've shared lessons learned through the projects and have contributed many of what have become best practices in managing communication system lifecycles. They've worked to build and maintain a trunked radio system open to use with end user radios for multiple vendors.

Jesse, we'd like to welcome you here today. Can you tell us a bit about the Phoenix Regional Wireless Network and share your background with it?

Jesse Cooper:  Sure. Back in 1999 the City of Phoenix and City of Mesa at the time partnered to build a region system. It was originally just the two of our jurisdictions that went forward with building out the regional network, but due to our fire partners, they have a automatic aid system where they dispatch 22 different jurisdictions out of a single dispatch center. So due to that need, we had to really build the system out to cover most of the metropolitan Maricopa County area. So we ended up building a pretty significant network with wide area coverage and lots of capacity. When we did that, after we implemented that system, started getting it on line post-9/11, many of the jurisdictions realized that interoperability was becoming a very critical issue and we started to expand our partnership with various jurisdictions. And since that time, since our implementation of the system in 2004, we've continued to add new partners, add sites, add capacity, and continue to expand our regional network across Maricopa County and we've even partnered with quite a few Federal, State, Tribal and County partners to really make a truly multi-jurisdictional, multi-disciplined regional network.

Ben:  Phoenix, Mesa, Tempe, and other partners in the Phoenix Regional Wireless Network have truly been pioneers in building shared radio systems. You said that the COPS 2007 Tech grant that Phoenix received helped the city and its partners move into a truly regional system with suitable governance. How are the City of Phoenix and its partners planning for the long-term management of the system and equipment replacements?

Jesse:  Well, with our 2007 COPS funding we expanded our capacity to allow new interoperability partners to join our system. With that we've added new members and new partners to the network. One of the first things that we've really tried to emphasize is the long-term management and the planning and equipment replacements that go with the large system. We've taken the regional approach, obviously, to share decision making. We have very high level of management officials that sit on our board. And our focus has really been on sharing the cost and sharing resources and additionally sharing our personnel. We quickly realized that with a system this large, one jurisdiction couldn't take care of all of the aspects of it. So in addition to the funding aspect of making sure we all share and contribute on a fair and equal basis, we've also really shared our expertise, our knowledge, and our personnel to manage the system.

Probably one of the first things we did for long-term management and planning was we really formed a true partnership with our vendors. Now a little bit different than just a standard customer/vendor relationship, where they provide a product and you're at the mercy of using whatever they produce, we've instituted a direction where we're true partners with the vendors.
The one thing that's allowed us to do is really steer them to meet our needs and partner with them to come up with a more structured process for how we're going to manage equipment replacements and system upgrades. And really kind of move forward with making sure we're in step with each other as to what can we produce funding-wise, what are our operational needs? What are our strategic plans? And how does that fit in with what the vendors are able to provide us?

So with that, we've had a roadmap where we know what the technology is looking like for one, two, five, even up to ten years. And with that we're trying to kind of come up with some financial planning so we don't have surprises for all our regional partners. We want to make sure they have adequate planning and advance notice that a system upgrade is coming or we need to replace some subscriber gear, so they're in a position to be able to forecast that accordingly.
Ben:  What other challenges are being faced?

Jesse:  I think not just unique to Phoenix or Arizona, but obviously the economy is causing us some uncertainty and some challenges that we're having to deal with. Some of the early planning... we had some very robust system enhancements. Some very, very large area expansions that were occurring. With the slowdown in growth, some of that planning has obviously had to be scaled back. So that's necessitated a little bit of... I don't want to say back-stepping. But we've had to kind of look back a little bit and say, "Do we really need, and what was proposed, and what the plan was?" So that's been some of our challenges and a few of the others have been just dealing with the various funding cycles and streams that all the jurisdictions are coming up with. Some people use a calendar year, other people use various fiscal years, some people have different bonding cycles and capabilities and capacities.

That's been a little bit of a challenge for us, to try and coordinate these ongoing system and equipment replacements with the various jurisdictions. Obviously when it was just the City of Phoenix or City of Mesa, it was a little bit easier for us to coordinate that. But now that we're up to the number of jurisdictions that we're dealing with, that's been a significant challenge.

One of the other challenges that we're facing are dedicated staff. We've quickly realized that this system and management, and replacement, inventorying, and long-term strategy really required some dedicated staff with a system this large. It's traditionally been people who did this as an extra component to their job. But we're quickly finding that you need somebody who's on top of this on a constant day-to-day basis to be able to manage that.

Additionally, a little bit of the challenges... we still have some jurisdictions that have not joined the regional system. So they're out there looking to build independent systems and the challenge that we face is we really want to at least propose to them what the options from our regional perspective are. Let them make a decision on which way they want to go.
Obviously if they choose a stand-alone system we'll work [with] them to integrate and coordinate interoperability. But we're still working with some jurisdictions that have opted to build stand-alones. That's been another challenge, is to how do we properly show them the benefits of joining a regional network and sell our regional concept to them?
Ben:  You spoke to it a bit in your last response, but in tough economic times it's often difficult to come up with needed funding for assistance. How do you work to convince the decision makers of funding needs for ongoing maintenance, upgrades, and replacements?

Jesse:  I think probably one of our chief benefits is we've been able to come up with some long-term strategies and planning and funding proposals that we've been able to present to our board. Our board is comprised of city management officials from each of the jurisdictions, so they do have considerable decision-making authority for their represented agency. But I think [it helps] by providing a solid roadmap at least as much as we can get them. Obviously one year is solid, two years is still pretty solid, as we move up to the three, four, five, ten, it's getting farther and farther out. We try to at least give them some of the information from a very high-level perspective of what are we looking at and what is it going to be in the future.

I think probably for our decision makers, one of the best things we've been able to do is to share success stories with them of how our system has been able to benefit the operational users. That has convinced them of the need to continue funding this emerging technology and to properly fund the technology as it moves forward.

I think by sharing some of our success stories—we've had, obviously, Super Bowl XLII, we've participated in TOPOFF 4, we have the NBA AllStar game this year—a couple other very large events that have occurred.

By highlighting the success stories of our regional system, I think that's convinced the decision makers in those positions of authority, that this is the way they need to go and they understand that the cost of maintenance, upgrades, and replacement ultimately provide a better public safety product for the public.

I think that's been key for us to be able to convince them to fund the projects that we're asking them to fund.
Ben:  With the number of users on your radio system and partners in its funding, you must have quite a variety of expectations from officials for more or less information on your roadmap for the future. How do you present system management and technology roadmaps to the level of detail that each need?

Jesse:  Well, our board members are obviously city managers or designees from their jurisdiction, so they have a different perspective on what they need from us. So when we do present those roadmaps or the system needs, we really start with a very, very high-level overview. Trying to put it in terms that they can understand of what is the end result and how is this going to affect police, fire, EMS, public works. What does this do for them? What do we gain? How does this help us move further on the Interoperability Continuum?

We really start with the high-level overview for those individuals, and what we found is we've set up additional meetings following that presentation where we get into detailed discussions and really involve their true decision makers or people who have to implement that decision. Where they can have a discussion over the impact, really drill down into the details of how did we come up with this information, offer their opinions and different strategies for how we're going to accomplish that.

Obviously, our board members are invited to attend those if they choose to, but we found that it's been successful to give them enough information to make an informed decision but offer the detailed discussion at a different venue so they can come in and offer their input.

I think with that, we also have a pretty robust operational working group where a lot of the line level and supervisory management personnel really sit down and vet all the information. We've been really fortunate that our board members trust the information and the decisions that are coming from that collective group.

So we've been the beneficiaries of them having a high level of trust for what we're recommending to them and really following along knowing that we've discussed all the right angles and the right decision on knowing what we need.
I think that that's been beneficial to us, and once again, just really focusing on the operational impact. We can talk about routers, switches, and towers all day, but that doesn't mean much to our elected officials or appointed officials. They want to know what is this going to get us, when we do have the major event or the large venue that occurs. How's our interoperability enhanced by providing these resources to us?

Ben:  What other type of questions do you face from officials regarding these roadmaps?

Jesse:  Well, I think because of their position, obviously one of the key components is the funding. How are we going to achieve that funding? How are we going to time our funding streams to be able to coordinate with each other? How do we time those? Really, probably one of the most critical components for our officials is really having to explain to them the need to enhance and upgrade our technology. A lot of our officials have been in place for many years, and they're used to traditional radio networks where we weren't approaching them a couple times a year asking for software upgrades or firmware upgrades to the radio equipment.

We've had to spend some time educating those officials as to [the fact that] these really are complex, technologically advanced networks [and] we really need to upgrade the system to obtain the latest benefits and features that come with that.
So that's been probably one of the key components for us. I think, again, highlighting success stories for us has been beneficial. The farther we get away from a major event, the easier it is to say no to something related to a system upgrade, so we just keep refreshing those success stories and how we're using this on a day-to-day basis.


When we do have large events or incidents that occur, we really try and provide them with an after-action [report] of this is how well the system worked and the reason we were able to respond the way we did was because of the system that we have in place.

For us, it's really encouraging them to understand and embrace the technology and understand the need, that it is going to be a more expensive proposition than traditional radio networks have been in the past.

Ben:  That communication, Jesse, does it go also down to operational personnel from different disciplines? So for instance, if the technology was used effectively across multiple jurisdictions and disciplines, do you share those success stories also with operational personnel?

Jesse:  Absolutely. Our operational working group, that really is the group that does the communications planning. Those are our COMLs [Communications Unit Leader] from the various jurisdictions. Those are the people that are out there making this interoperability happen. Typically, our operations group are the ones who prepare those after-action reports, sit down and talk about lessons learned, improvement plans. How are we going to make interoperability function better for future events?
So we really do have a very large group. We even invite the operational folks who are not part of our regional system to sit in on those and participate with us if they choose to. They may have had a very slight role with this, but we invite anybody and everybody from the region who may have had a response function.

Even if they haven't, if they're going to be participating with us in future events, we really encourage them to sit down with us and work through those. We really get down to the details at the operational level and then produce that information higher up so they understand the success of that, as we go to those elected officials or those appointed officials.
Ben:  Communications and information sharing systems are fundamentally dependent on the technology that vendors supply. What challenges do you face getting technology and project roadmaps from vendors?

Jesse:  Well, I think probably one of the first challenges we had to overcome with several of our vendors was just the proprietary nature of the information that we were asking for. There's plenty of trade shows and other events where they unveil their new products, but really, looking for a long-term strategy or long-term roadmap for planning, both from a financial or technical perspective, we were really asking for some pretty detailed information that could compromise our competitive advantage in the marketplace. We really had to sit down and build that level of trust and again, that goes back to my earlier discussion of it's a partnership, not the standard customer/vendor relationship that we're typically used to.
So with that, we had to sit down and work out some of those details. They're going to give us a lot of information that could do quite a bit of harm to their organization if we were to release that. We had to enter into some nondisclosure agreements and make some commitments as to what we would share with our partners or what we could share or how they need to enter into a nondisclosure agreement.

That's been a little bit of a challenge. Most vendors are very closed about future plans and releases and where they're heading with this continuing product and entering some new product, so we've really had to work with them on that. I think one thing that we quickly realize is we are asking for long-term roadmaps from our vendors but they even have issues with their vendors, their third-party providers, as to trying to obtain that roadmap information from the people who supply them some of their accessories or some of their computer equipment.

So it's not only us dealing with the vendor directly, it's understanding that they also have to deal with their suppliers and they may be getting fuzzy information or not real solid detailed planning from their vendors. So we have to accept that a little bit when they came to us and said, "We're not positive on something because our third-party vendor X could not give us a timeline on when that was going to occur." I think that's been another challenge that we've had to face and really just accepting it and understanding it.

The uncertainty of technology has also caused some concerns. They're willing to give you very solid information for a year, two years, even up to three years. But anything beyond that—I think given the way technology is advancing, and equipment's being phased out and changed—that has added to some of our uncertainty with the information that we have been able to provide our elected officials.

Changing mandates from the FCC—some of those are debatable—as to "Are those dates firm or are those going to change? Are they going to shift any?" Some of that information that we are getting, we are basing on standards and mandates that are in place today, but that could change. I think with the evolution of the P25 standards, some of the changing mandates and timelines, I think that's been another challenge that we have had to really work with the vendors to understand where they are coming from. They're basing their information on what they have today but they've been very clear that that is subject to change the further out we get. That's something which we've accepted and we understand but it is a challenge that we have to understand and really accept as we move forward with the roadmapping.

Ben:  What additional information would be useful for your own lifecycle planning needs?

Jesse:  One of the challenges we've really had is the technology roadmaps are fairly solid that we're getting from a lot of the vendors. One thing that I know we would really like to see is a little firmer fiscal commitment on their part. But again, we understand that we don't know what new routers are going to come out [in] five years. What are the costs of those going to be? What's going to be the current market price for some of that? A lot of the information we receive comes in ranges. High range or a low range, based on today's dollars and obviously trying to account for any type of inflation that may occur. We do provide ranges to our officials as to, "This is what we think something is going to cost," obviously not knowing where the economy is going to go, or what the cost of equipment's going to be. We would like to feel a bit firmer fiscal commitments on those, just to help us with our planning phases.

Better control from the third party vendors. I think they're doing the best they can with trying to control some of their roadmapping and some of the information they're receiving from their suppliers. But I think with the globalization and products coming from all over the world, it's even difficult for the vendors to control some of that information on some of that product that is coming to them and that, in turn, results in less firm commitments for us in our roadmapping.

So as far as lifecycle planning goes, it would be really nice if they could give us some better information. Again, we do understand that they're at the mercy of their suppliers, just as we're at the mercy of them to tell us what the roadmapping is. But those two things would be helpful. But I think with any system development lifecycle planning, you always have those uncertainties that you have to accept as a cost of doing business with this planning.

Ben:  Regarding system lifecycle planning, what core competencies do you see in Phoenix staff that are necessary to enable you to be able to adapt to the market changes and the market realities and uncertainty?

Jesse:  Well, I think our infrastructure support folks have to be competent on the emerging technology, the changes that are occurring, and how they handle these unexpected changes that we get. There are a lot of times where the vendors will report back to us that a product that we rely upon, that is a core component of our system, is being discontinued and that's put quite a bit of stress on our technical staff and our support staff to have to quickly adapt and work with the vendor—not against the vendor—but they have to come up with the mutually agreed-upon solution. So it not only puts the vendor into buying and to come up with a solution, but it's requiring some pretty creative engineering on our people’s part to quickly identify alternative solutions, work with the vendor, and come up with those. We're all having challenges finding qualified personnel to work in this world that we know is the radio business now. One of the challenges we have really had is making sure our people are trained and are able to [come up to] speed on the ever-emerging technology and that has been a challenge for us. We've really tried to spend a lot of time educating, sending our people to the appropriate training. But the technology is moving so fast that it's difficult for us to make sure that they have the appropriate training to be able to handle those curve balls that come their way.


Ben:  Has executive leadership been supportive of that ongoing training?

Jesse:  The training—we've tried to incorporate that into our overall fee structure and as we get new technology, we build in the initial training and pretty much the ongoing training. I think just like most public jurisdictions around the country, training has been one of those things that has suffered early on in the budget-cut process. But what we've really tried to do with our executive leadership is explain to them how important [it is] having the knowledge, the ability, and the experience to handle these changes quickly and for such a mission-critical system. They have been supportive, and I think one thing that we have really been trying to build in through our fee structure is to make sure that we are recovering enough cost to train technicians, properly equip them with the right tools and measurement equipment to be able to do their job effectively. Our executive leadership has been extremely supportive of that. Every year we revisit our operations and maintenance fee structure and that's one thing that we always try and include to make sure that we have sufficient funding for the proper training and equipping of our personnel. So far everyone's been supportive about it, I think, as they understand the technology, and how critical it is that we have properly trained personnel to handle that. It's been easier to try and make sure we incorporate that into our fee structure.

Ben:  Phoenix and its partners have been open about using technology from multiple vendors. What additional lifecycle planning challenges do you face due to that?

Jesse:  Well, from the infrastructure side, a significant number of vendors supporting infrastructure are still kind of a ways out. There's a couple of vendors that we handle on that but I think where we really had to adjust our plans had been from the subscriber end of things. We’ve had a number of new subscribers come on the market. There had been vendors really wanting to participate and work on our system—we're open to that. We offer that to basically anybody who wants to bring equipment out, test it, validate it on our system; we are more than happy to work with them and really test the equipment. One of the challenges that we had from the subscriber vendors is that how are they going to keep up with the infrastructure changes that occur. I think that what we've discovered is extremely important: that the relationship between the vendors can really impact the performance that we see as an end user. By that, I mean if the relationship is really good between an infrastructure vendor and a subscriber vendor, we've been able to really see the benefits of how well that equipment will work on our system. If that relationship isn't as strong or it's a new player in the arena, we've really seen where their equipment can struggle.

So from our perspective, some of those challenges have been we can't control the relationships between the various vendors but we do encourage them to communicate with each other, work through a lot of those issues, and really come up with the right solution.
But again, not only are you dealing with the infrastructure lifecycle, you're dealing with subscriber lifecycles. With multiple vendors out there handling subscriber gear, it's becoming a full-time job coordinating those different roadmaps, putting those together. Obviously we want to be sensitive to the competitive needs of each of them, so we have to be very cognizant to make sure that we're not cross-sharing information that we really shouldn't be.
Some additional challenges we've faced—or really, if we do have an issue with a couple of different vendors—is who's really at fault. I don't mean “fault” in a bad way, but who's job is it to resolve the issue? We have noticed in a multi-vendor environment that there can be some finger-pointing that occurs. That's where I think those relationships really come into play, where if the two vendors have a great working relationship, they're willing to partner together and solve those issues for us. And we've had some relationships that really aren't there and that's caused some grief trying to get them together to try and solve those problems.

So I think for us that's an additional planning challenge that we really have to just be aware of. Again, it's managing the relationships while working through the technology issues. And those can be just as important, if not more important, than the nuts and bolts of the technology alone.

Ben:  Continuing with the theme of challenges, Phoenix has been an earlier adopter of digital trunked radio technology for shared system, a testbed for emerging products. What have been some of the benefits and additional challenges of being in this position?

Jesse:  I think we've been able to really explore a lot of the new features and a lot of the new technology. And by being a test system for each of those vendors, I think we've been in a unique position to help direct some of that new technology and really offer some of our opinions and desires and operational needs that have really changed the way some of those vendors have produced some of their products. Obviously the Phoenix metro area has some unique challenges that don't exist in other parts of the country, and they have unique challenges that we don't have to deal with. But I think that as the vendors go to the various systems around the country and test their product, it's our hope that we've been able to give them some information that may benefit not just us locally but everyone who's going to end up using that product. So it's been very exciting for us to be able to help drive some of that new technology and really direct where some of the new features and enhancements are going.

A challenge with that has been [that] it is a time-consuming venture. While we have made that offer to basically any vendor who wants to come out and participate and work with us, with that has also been the issue of working with those vendors to help solve some of their issues and work through some of their challenges that they've faced. That can be frustrating at times. It can be difficult when you're trying to get your regular job done. But it is a commitment that we've made to those vendors and ultimately it benefits everybody in public safety for us to spend the time with them.

As much as I say that a challenge has been the time involved, I think ultimately that has helped establish some better relationships with those vendors. Working side by side with each other to solve some of the issues and offer enhancements, and really discuss their products and how they're functioning, we have really built some strong relationships with multiple vendors out there.

Even if we don't end up using their subscriber gear, for example, we still keep in touch with them, we still offer them some advice. They may end up producing something in the future that does work for us. So we really keep those relationships strong. And ultimately one day we may end up doing business with them even if we can't work something out today.
So along with the challenge, I think it's produced some unique benefits for us to be able to work with some great people from all over the world to try and produce some of the public safety equipment that may be used by some jurisdictions that don't have the opportunity to work with them in the early developmental stages.

Ben:  Final question. What do you see is the impact of multiband radios in systems such as yours?

Jesse:  I think multiband radios are going to offer us some unique capabilities that we've never seen before. But at the same time I think they're going to offer us some different challenges that we're going to have to overcome and really work through. In our region we have various spectrum and use by all kinds of different agencies. We have VHF, UHF, 700/800, trunked, conventional. So we do have a lot of different users, and a multiband radio is going to be able to meet a lot of those needs. We are one of the DHS pilot sites, so we're looking forward to testing some of those radios and really working through how they're going to function on our system.

I think from our perspective they're going to work tremendously well in our cache environment. We have some All Hazard Incident Management Teams that are deployed in various locations and have to work in various frequency bands, and having those radios available for them is going to be a tremendous asset. Additionally, we work with a lot of federal partners due to our proximity to the border. And I think the multiband radio is really going to offer us some capabilities to enhance our interoperability opportunities with others.

An impact of that is going to be the coordination that is required. With a radio that has such robust capabilities and abilities to communicate across spectrum, we've already even in the pilot phase started to see some of the unique challenges that are coming with coordination. If the radio can communicate on 15 or 20 different agency bands or frequencies, how do we coordinate agreements? What's the operational impact of having 3,800 Phoenix PD users with the capability of talking on a jurisdiction that's used to handling 10 officers?

So with those, it's not just the technology planning that we're having to face, but it's also the operational component of what does this mean, having a radio that's out in the field that is capable of talking to so many different jurisdictions. Additionally, how do we train our personnel who have this new capability to be able to use it responsibly and not overwhelm other jurisdictions?

So I think the impact is going to be positive. I think it is going to necessitate some additional training, some coordination among technical staff and operational staff that may not have been present before. But again, I think that's the relationship component, everybody getting together, working through these solutions to prove the end user's interoperability out on the street.

Ben:  We'd like to thank our guest, Jesse Cooper, Communications and Information Technology Project Manager for the City of Phoenix. Jesse, thank you very much for your time.

Jesse:  Thank you for having me.

Ben:  You've been listening to a SEARCH podcast on managing communication system lifecycles in the multivendor environment. This project was supported by a cooperative agreement number 2007-CK-WX-K002 with the U.S. Department of Justice Office of Community Oriented Policing Services. Points of view or opinions expressed in this podcast are those of the guest and moderators and do not necessarily represent the official position or policies of the U.S. Department of Justice. 
[music]
p.