Brian MacKenzie

Online Systems Engineering Manager

2K Games / Take-Two Interactive Software, Inc. - Novato, CA / New York, NY - Sep 2014 – Present

• Took an opportunity to relocate to the California headquarters of 2K Games, a subsidiary of Take-Two Interactive, to found the 2K Online Engineering team. The responsibilities were similar to my role at Take-Two, although specifically focused only on 2K Game titles. I hired, developed talent and managed a team of administrators responsible for the online game, presence/telemetry, and web property hosting needs of 2K’s various titles. Online Engineering acted as an internal managed service provider, where I managed relationships with various internal clients/developers and fulfilled the technology infrastructure needs for their individual titles within our global datacenters as well as various cloud providers. Our team was responsible for systems provisioning, scaling, solution design and implementation, performance and process optimization, patching, monitoring/alerting, and breakfix issue resolution.
• Successfully launched or assisted in the launch of online components supporting: NBA 2K16 (2K Sports), WWE 2K16 (2K Sports) Civilization: Beyond Earth (Firaxis / My2K), XCOM 2 (Firaxis / My2K), NBA 2K15 (2K Sports), WWE 2K15 (2K Sports), Evolve (2K Games / My2K), Grand Theft Auto V (PS4 / XB1) (Rockstar Games)
• Administered thousands of physical and virtual servers, representing various title's production, qa, demo, and staging environments. The majority of servers under our purview were CentOS 6 and virtualized across multiple VMware ESX clusters
• Led weekly internal client infrastructure standup meetings, where we discussed issues and work completed within the last week as well as any new projects or other relevant items
• Maintained server configuration management via Puppet to ensure server consistency, appropriate server hardening, and to facilitate provisioning efficiency
• Led a project to stand up, hire, and develop talent for a global internal tier 1 NOC team, which would handle application monitoring, incident response, and off hours work for our team and various internal clients we service
• Implemented Slack chat as a chatops/communication platform for our team and the internal clients we support. Set up various channels and service integrations to streamline issue and operational task visibility
• Implemented a kanban board project tracking system for our team and clients to organize and raise visibility on task status and blocking issues, as well as to track time associated with various clients and projects.
• Led a project to migrate internal server and application monitoring to a new system (Nagios-like) to provide service alerts for outages, as well as historical graphs used for environment trending
• Implemented external monitoring of service endpoints within Pingdom and Thousand Eyes to obtain greater visibility into the scope and impact of network and application issues
• Implemented formal team on-call rotation via PagerDuty, ingesting alerts from various monitoring platforms
• Coordinated with internal security teams to identify and remediate various security vulnerabilities in our environment (Shellshock, POODLE, Heartbleed, etc.) as well as to investigate and remediate malicious behavior patterns, such as DDOS attempts
• Led a project to migrate primary resolution of DNS for all externally accessible DNS endpoints to a third party provider (thousands of addresses)

Sr. Director Network and Systems Administration

Stevens Institute Of Technology - Hoboken, NJ - January 2014 - September 2014

• Managed the systems and networking administration teams who directly administered the physical infrastructure supporting the day to day operation of the University
• Led the project to entirely redesign and replace all networking devices across campus:
  • Increasing stability by introducing redundant hardware in the core and distribution layers
  • Increasing availability and capacity in buildings and across the campus by more than doubling the number of access points deployed
  • Supporting BYOD by implementing equipment that facilitated easier on-boarding
  • Provided a simple and user friendly VPN client deployment mechanism
  • Worked directly with procurement on the creation of an RFP submitted to multiple vendors for bid, acting as the primary technology contact for on site tours and vendor questions, as well as analyzing bid results and selecting a vendor. Also acted as the primary business contact for the chosen vendor during their fulfillment of the project.
• Led the project to redesign the physical fiber cabling infrastructure across campus:
  • Increasing stability by adding redundant fiber in the access, distribution, and core layers of our network architecture
  • Increasing the speed from the core to many buildings from 1Gbps to 10Gbps
  • Adding resiliency via the creation of a fiber ring
  • This involved working directly with procurement on the creation of an RFP which was submitted to multiple vendors for bid, acting as the primary technology contact for on site tours and vendor questions, as well as analyzing bid results and selecting a vendor. Also acted as the primary business contact for the chosen vendor during their fulfillment of the project.
• Led the project to redesign Steven’s physical datacenter; requirements included scaling to handle an increase in power and cooling utilization, a change in the physical footprint, and addressing other environmental concerns. This involved working directly with procurement on the creation of an RFP which was submitted to multiple vendors for bid, acting as the primary technology contact for on site tours and vendor questions, as well as analyzing bid results and selecting a vendor.
• Handled the primary relationship with 3rd party technology providers as well as various consultants
• Coordinated regular network technology design sessions with my teams to analyze strengths and weaknesses of all vendors in scope and hold roundtable discussions with other universities about their experiences and recommendations.
• Led the project design and implementation of a physical Microsoft Exchange e-mail server with assistance from a technology consultancy
• Worked with a technology consultancy to migrate DNS and DHCP services from Linux to Microsoft domain servers
• Maintained balanced budget documentation for multiple million dollar state funded technology grants
• Participated in a design team to implement physical security camera’s across the campus
• Collaborated with the web technology team to scope out and perform vendor analysis for a web accessible collaborative local file sharing system

Sr. Manager DevOps

Thomson Reuters - Hoboken, NJ - September 2013 - January 2014

• Managed a group of system administrators that administered the product's qa, demo, and staging environments
• Scaled multiple production, qa, demo, and staging Citrix Presentation Server/XenApp and SpringSource TC Server/Tomcat farms which support over 850 applications and 1,600 Fortune 500/1000 tier clients with over 26,000 global users
• Developed and maintained configuration and application management solutions to ensure server consistency and appropriate server hardening
• Handled primary relationship with internal and 3rd party datacenter providers as well as various technology consultants
• Worked directly with developers to define software requirements as well as to implement custom software solutions
• Assisted sales team by performing pre-client datacenter tours, participating in security and project scoping calls, as well as designing custom hosting solutions and environment SLAs
• Handled primary relationship with 3rd party auditors to perform companies' yearly SSAE16 SOC-1 and SOC-2 audits
• Coordinated with 3rd party security vendors to facilitate yearly network and software vulnerability scans as well as issue remediation plans

Technical Operations Manager

Quality Health - Jersey City, NJ - November 2011 - September 2013

• Administered multiple consumer facing websites that that handle over 3 million unique visitors and 14 million pageviews per month
• Managed a team of system administrators that manage our 60+ physical and virtual server production, staging, and development environments as well as all internal IT operations
• Reverse engineered, rebuilt, and migrated the Production and Staging environments to new hardware in a new datacenter.
• Built a DR environment in a separate datacenter, which includes Production content replication mechanisms
• Administered Production webservers running Apache and Jboss on RHEL 4 & 5
• Initially created and maintained server build templates for our Production, Staging, and DR environments using Puppet to ensure server consistency and appropriate server hardening
• Successfully migrated all of our Git repositories an alternate server and assisted users in updating their remote links
• Set up full Git repository offsite backups for DR
• Assisted developers and site content generation users with various Git repository commit issues
• Maintained close to 200 production, staging and development scheduled jobs spread across 6 Jenkins servers
• Handled primary relationship with 3rd party datacenter and DBA support providers
• Administered Windows Server 2008 servers running multi-site Active Directory implementations, Group Policy, DNS, and DHCP services
• Implemented environment monitoring via AppFirst to provide service alerts for outages, as well as historical graphs used for environment trending
• Interfaced with vendors for technology purchasing (Dell, CDW, RedHat, Akamai, etc.)

Sr. Manager Network Systems

Thomson Reuters - Hoboken, NJ - February 2008 - November 2011

• Served as a member of our business' Executive Committee where I presented monthly to various levels of upper management on the state of our production environment, which includes statistical analysis of concurrent usage and projected growth
• Managed a group of four system administrators that administer our product's 140+ server production, demo, and staging environments
• Scaled and administered multiple production Citrix Presentation Server/XenApp 4.5 farms which supported over 850 applications and 930 Fortune 500 clients with over 9500 global users
• Administered production webservers running Apache/Tomcat and SpringSource TCServer on RHEL 4 & 5
• Administered production webservers running IIS 6 & 7 on Windows Server 2003 & 2008
• Led project to integrate our Citrix environment with a PingFederate Single Sign-On implementation to further integrate our hosted offering into a suite of similar products
• Led a project to migrate the data from our production Oracle 9i RAC to a newly implemented Oracle 10gR2 RAC
• Led projects to define disaster recovery procedures and implement those procedures with recovery tests
• Implemented a Nagios environment monitoring system to provide service alerts for outages as well as historical graphs used for environment trending
• Implemented a database of IT procedures, server configuration and security details to ease the completion of sales' pre-client RFI documents
• Worked directly with development to define software requirements and implement custom software solutions for clients or ease of administrative tasks
• Handled primary relationship with 3rd party datacenter provider
• Coordinated with 3rd party vendor to perform yearly software and network penetration tests and work with vendor to remediate any findings
• Coordinated with clients to resolve production hosting issues and facilitate client initiated penetration tests
• Coordinated with external auditors to perform yearly SAS70 & Financial audits

Director Of Information Technology

TaxStream, LLC (Acquired By Thomson Reuters) - Hoboken, NJ - January 2006 - February 2008

• Served as the initial IT employee in a sub 30 person small business which went on to grow to over 125 people and obtain a rank of 51 on Inc. 500 magazine's 500 Fastest Growing Private Companies list prior to it's acquisition by Thomson Reuters
• Managed a group of three system administrators that managed all internal IT operations as well as our client facing production environment
• Designed, implemented, scaled and administered multiple production Citrix Presentation Server/XenApp 4.0 farms which now supports over 850 applications and 930 Fortune 500 clients with over 9500 global users
• Administered production webservers running Apache/Tomcat and/or JBoss on RHEL 4.
• Designed and implemented a multi-office voice and data WAN using Cisco 1801 routers
• Administered Windows Server 2003 servers running Active Directory, Group Policy, DNS, and DHCP services
• Led a project to upgrade our Windows Small Business Server 2003 domain controller to Windows Server 2003 Standard
• Implemented and administered SugarCRM for use as sales pipeline and support ticketing solution
• Administered Nortel BCM and Callpilot hardware for use as a phone network, voicemail, and call center solution
• Used Microsoft RIS to image and deploy more than 150 laptops and desktops
• Administered internal Oracle 9i and 10gR2 servers running on RHEL 4
• Interfaced with vendors for technology purchasing (Dell, CDW, Verizon, etc.)
• Coordinated with clients to resolve production hosting issues and facilitate client initiated penetration tests
• Coordinated with external auditors to perform yearly SAS70 & Financial audits

Client Consultant

SunGard Trading And Risk Systems - New York, NY - July 2004 - January 2006

• Implemented individually tailored software installations and upgrades in client's production environments
• Acted as a client contact for software, database, and environmental issues
• Administered internal servers running Windows 2003 Server, Solaris 8, SQL Server 2000, Sybase ASE 12.5, Oracle 8i, and VMWare GSX server
• Created and maintained over 60 virtual machines used for testing software in client replicated environments
• Investigated and corrected client database issues caused by upgrades or software glitches, using SQL
• Created and configured IIS Virtual Directories to implement ASP applications
• Coordinated VPN setup to remotely troubleshoot client's internal issues using Remote Desktop, PCAnywhere, Checkpoint and Remote Administrator

Technical Support Specialist

Securitas Inc. - Parsippany, NJ - February 2004 - July 2004

• Fielded technical calls from more than 600 offices nationwide
• Replaced and installed hardware and software applications on company workstations
• Troubleshooted custom applications running on OS/390, DOS, SCO UNIX, Windows 95, 98, 2000, and XP.
• Remotely analyzed and repaired company workstations with LANDesk Management Suite 6.6
• Was responsible for handling orders of new/replacement parts from Dell

Web Developer

Marist College - Poughkeepsie, NY - September 2001 - December 2003

• Developed web applications using PHP and SQL
• Created and modified web documents on the Marist College website using HTML/XHTML, CSS, and JavaScript
• Maintained development and production versions of web applications and documents on a RedHat Linux platform
• Responsible for development of both the School of Communications and The Arts as well as the School of Graduate and Continuing Education's websites
• Headed a team that was responsible for recoding large portions of the Marist College website to meet the W3C's XHTML 1.0 Strict standard
• Worked with custom applications running on an OS/390 platform

Marist College, Poughkeepsie, NY

Major: B.S. Information Technology - 2003