This is a follow-up from my earlier post on how hard it is to run an ISP business. I would have preferred a single article but the first article was long enough, for my liking.

In my earlier article, I went into detail and highlighted the pains and costs we had to deal with when running an ISP business and advised readers against starting one. But at the end of the article, I conveniently pointed out that not only were we profitable, but we also became profitable faster (in 2 years) than the industry average (in our country it’s 4-5 years). Considering that most ISPs are not even profitable ever, they just hang on to it, because to them it’s just a small part of their other more profitable business ventures. But by doing so they massively undersell the market, without making any money, just to stay relevant.

Do It Yourself

One of the best decisions we made when we started was doing most of the networking, operations, and support ourselves instead of hiring employees. We taught ourselves the basics of configuring Cisco Routers and Switches, Mikrotik CCRs, the fundamentals of planning and setting up fiber layout, how signal strength affects service quality, how to find fiber breakage, following best practices. We researched and learned all these by simply using Google.

This is highly not advisable. For the first 6-7 months, we were putting in 12-15 hours a day almost every day. It was a very stressful time but it was also a lot of fun, we were learning and fixing and improving things on our own. We made a lot of really silly and expensive mistakes but it also made us wiser and helped us make better decisions down the road.

The alternative would be to hire very expensive, certified, knowledgeable, and qualified employees. I am not advising against it, it’s the right way to go. Hire people who know what they are doing. But usually, employees will also want to go by the book, they won’t want to go through complicated loops and compromise in tools and hardware to save money for the company. We know this because we were once employees too. There is nothing wrong with this type of thinking, but it won’t work for us. We needed people who can put in the hours and go the extra mile to save money for the company in the short-term until we can stand on our own feet. The only people who were willing to do this was ourselves.

We weren’t exactly computer illiterate, I come from Linux sysadmin background, mostly self-taught, so it was a natural transition for me to learn things by myself and RTFM. The DNS server I deployed (my first), is still working happily, even though our traffic increased by more than 500 times since I set it up. I also worked on our MRTG, NMS (Nagios, librenms, smokeping). Scripts to auto-scan our IPs against popular blacklists, made a custom dashboard to get an overview of our network status. My partner mostly worked on configuring the routers and the switches and I helped when I could, there was a lot of trial and error in the beginning.

On the Operations side (planning out and executing fiber layout), we did have an outside, on the phone, consultant. We learned from him enough in the first 4 months that we were able to train one of our own employees to take on the responsibility on a full-time basis.

In the first three months, we didn’t have any support staff. We used to keep the office phone number with us (call forwarding) and divide time between us to take on phone calls from customers. This meant we were always on call but this also helped us learn the nature of clients’ complain hands-on and to deal with clients and fix their issues. Hiring a support person would have insulated us from understanding a lot of problems clients might face.

Now, most of these responsibilities are maintained and run by dedicated and knowledgeable teams of employees, but having worked on these problems in the early stage of our company gave us a deeper understanding of day-to-day issues and how to quickly fix them.

Cache / Peer as much as possible

As mentioned in the previous article, getting GGC saved us massive amount of money. At one point, 70% of our bandwidth was YouTube traffic. I tried to dabble with Squid as a transparent, forward proxy to save some bandwidth but soon abandoned the project. It’s one thing to use squid with a couple of hundred users, but it doesn’t scale well without investing on enterprise-grade hardware and full-time tweaking with squid configs. So we did exactly that, about six months ago we partnered with a caching software provider and invested on an enterprise-grade hardware solution it is still the single most expensive hardware in our server room (after fully loaded with near-line SAS Drives, DDR-4 ECC RAM, and rare transparent 10g sfp+ bypass network cards). I can’t name the cache software vendor (NDA and all), but after the initial hiccups, we are getting about 30% BW saving on non-SSL traffic. The software does torrent caching so that helps a lot too. We do profit-sharing with the vendor, so it’s in their best interest to increase the caching performance.

We also aggressively peer with as many upstream as we can, initially, it was about reducing latency but now that our total bandwidth has increased by many orders, it also helps to cut our BW costs. We are also looking into Akamai and Netflix on-premise CDN to save more BW.

All that work and no money to take home

The first year I and my partner took in zero money. It was a very difficult choice for us because we weren’t even sure this business will ever make any money and for the longest time, we were sure we would eventually have to sell the company and cut our loss. For me, I was less concerned about money and more bummed out about the lost time, the time I could have spent doing something else.

On the hiring side, we were only hiring people we absolutely needed. No employee in our company can say that they have nothing to do, it’s not like we are over-working our employees. Most of our employees have very specific responsibility and they are held accountable (privately) for any screwups. We also have an open-door policy, meaning any employee at any time can just walk into my (or my partner’s) office or call us anytime with any problem. This meant we were able to deal with issues sooner before they got out of control and thus saving us money and headaches down the road. We know that this policy won’t scale well when we grow bigger, but for the time being its working pretty good for us.

Do start an ISP business if you don’t mind the stress.

I am not saying following our steps will work out the same way for you too. We had our share of a healthy dose of dumb luck. We are not exactly swimming in money. We are re-investing most of our little profit back to the business anyways. But the fact that we are profitable at all and within two years, in this sector, is a miracle to us and something we are very proud of.