Track Your Internet Quality for Free - Netprobe Full Tutorial

0:01 Networking security and Wireshark. If you're someone who suffers from chronic internet access issues at home and are looking for a simple and effective way of monitoring your internet access quality 24x7 in order to: one, understand where your problems are; and two, keep your ISP honest and potentially switch ISPs, Net Probe was written for you. So, I had this exact same challenge in my own home network and professionally. There are thousands of tools that do exactly what Net Probe does, which is basically to monitor the internet quality or the network quality of your ISP. However, I couldn't find a suitable tool that was one, free and open source; two, simple; and three,

0:49 allowed you to collect as much data as you want over time so you can compare over long time frames. And that's exactly what Net Probe does. So, this video, we are going to go into how to install and set up Net Probe on a PC or an old server in your home network, how to navigate GitHub for those of you who aren't really familiar with Git, and really just how to read the data, how to go through each one of these metrics, and how to use it for your own purposes at home. So let's get started. All right, so before we get started, as always, a little bit of theory on how Net Probe works. So, to get started, you want to set up Net Probe on a PC, Mac, or Linux machine that's directly connected to your ISP router via an Ethernet cable. Okay, you don't

1:37 want to use wireless for this. You know, an old kind of machine you've turned into a Linux server is good for this, an old PC you're not really using, or even your primary PC, granted that it's always online, because you want this to be running 24/7, so that it's constantly testing your ISP performance. And that's also in the GitHub readme if you take a look through it. And what Net Probe is doing is fairly simple. From that PC, where you install the software, on a regular basis, Net Probe is sending out probes, as you'd imagine, through your ISP network to various destinations, either DNS servers or public websites. So, in the case of public websites, we're using the probe,

2:22 quote unquote, is ICMP, it's ping, and ping allows you to send some traffic through the router, through the ISP network, and then over the public internet to a series of public websites. Now, this isn't just one server obviously, we use multiple websites that we're testing, so if you look at it that way, you're probably looking at something like this, where you have a destination here, destination here, destination here to different parts of the internet, which do not reside on the same actual networks. And the reason we do this is because we want some diversity, such so that the only common element in those connections is really the connection through your ISP network to the internet. Okay, so we do ping tests to a variety of different well-known public websites,

3:12 which I'll show you in the config file. We record back the latency and the loss detected, and then we average those values together. And what that allows you to do is get a sense of the performance of this leg here, because if, for example, we only tested one public website, well, and you saw issues, those issues could be here, or the issues could actually be here, right on the actual network that that public website is on. So, by increasing the number of targets, we increase the diversity and the statistical probability that the issues we detect are actually going to be within the common leg, which is my on-ramp to the internet, and not necessarily the internet itself. Okay, so that's the first piece. With the DNS test we do in Net Probe, we do the same thing. Right

3:59 from the same software, we send out DNS queries to various DNS servers online. Now, the DNS test is not here to test your ISP unless you use your ISP's DNS server as one of the destinations, which I'll show you how to do. But it's there to just measure relative DNS performance, so that you can compare, say, the DNS offered by your ISP to, say like, the ISP itself has its own DNS that it gives you, but you also want to see how that compares against like Google DNS and like you know, Quad9 DNS, things like that. That's what the DNS test is there for. But that's effectively it, right? It's a tool that is sending out on a regular basis, pings to various public websites, ICMP ping,

4:47 measuring latency and loss, the latency allows it to measure jitter because jitter is just a rate of change of latency, and we also do DNS tests. But that's effectively it, and that's really why you want to have a physical connection here because really, you want to make sure that if there's any problems, it's not, you minimize the possibility of the problem is here. Okay, if you use wireless here, chances are you're going to see all sorts of packet loss and latency and jitter, but that's not going to be because of this leg over here, likely it's going to be because your wireless has interference, and there's microwaves in your house and other things. So that's effectively it, we're sending these probes out on a regular basis to both websites, DNS servers, collecting the data, and then Net Probe, the software, is actually aggregating all that data into a

5:33 database and allowing you to build nice visuals so you can track that performance over time. So, let's switch gears now to the more practical side and show you how to install Net Probe. Now, the repo, or repository, that has the code for Net Probe is posted in the show description. I also posted it in some previous videos as well. The code and the application are open source; there's an open source license I throw in there. So, feel free to use it for personal use, for use even if you're a small business, I don't mind you using it. If you need to fork it, you want to modify the code, go right ahead, it's all yours. Now, if you're already familiar with GitHub and Git, I'm not going to cover how to clone the repo; you can look up lots of different tutorials

6:22 on YouTube on how to do that, but I'm assuming you know how to do that already. Really, you just get the link, clone it, which means just copy it, or make a local G copy to get on your machine, and then you get all the source code downloaded. If you're not familiar, then you can just simply go and download the code to a zip file, and then once it's in a zip file, you can extract it to a folder locally on your machine, and then you have the files that look like this. So, that is step number one, is just getting the source code downloaded to your machine. It does run on Python. However, because this is a containerized application in Docker, which I'll get to next, it doesn't matter that you don't have Python installed. Now, it does help if you want to be editing some of the config

7:10 files that you have something like VS Studio U, or even Notepad++, just to edit the code. If you don't, that's fine, it's not a big deal. You can just deal with Noad. Okay, the second piece, after you've downloaded the code, is Docker. Now, this may give some of you some anxiety, especially if you haven't used Docker before, or you hear a lot of developers talking, and you know, developer jargon, speak about Docker. Docker is a tool that allows developers, or just enthusiasts to code, to deploy containerized applications. Containerized applications are applications which, cloud applications are, or well, cloud applications are containerized applications, put it that way. They're really applications that can be deployed to a variety of different

7:56 environments and operating systems and things like that, without really worrying what the underlying operating system is, and that's what the benefit is here for Net Probe. The benefit is that, with Docker installed on the host machine, so the PC that you're going to be setting this up on, it doesn't matter if it is Linux, Windows, or Mac, as long as you have Docker running, you can run the software, you don't need to have any local packages or modules on your machine, it's all self-contained. Okay, now installing Docker is actually super simple. Again, I will post a video in the description, showing people how to actually install Docker, cuz there's probably a hundred tutorials out there on the internet about how to do that, but so, I'm not going to cover that right now, but what I want to cover is that once you've installed

8:43 Docker, if you're doing it on Mac or Windows, you would get a Docker desktop GUI that looks like this. And really, what you need to understand is whether or not it's running, and in this case, Docker is running because it is on. I don't have any applications running in Docker yet, and we're going to show you how to do that with the code, but the first step is just to make sure that Docker is installed and running successfully. Believe it or not, we are literally, you know, 30 seconds away from actually being able to launch and use the application. That's how simple it is. But before we get to it, I just want to explain some of the configuration, just show you a little bit behind the scenes. Okay, the code is here; I've just opened it up in VSS code. Again, you can open it in Notepad; I've done a lot of development in Notepad because really, there's

9:31 not really much you have to modify. Net Probe was designed to just sort of be set and forget, and let it run, and then collect data, and then you can review the data in a week or whatever you want. But the one file I want to point out is the ENV file. Right, so if you're a developer, you know that the ENV file is usually used to store variables, right, usually private variables, but in this case, I made it public because ultimately it's just configuration. And there's some details in here that I wanted to go over real quick. The unique IDs up here are not being used, at least in the front end. The next thing that's relevant is the sites, right. So, I mentioned in the initial theory section, with the ping test, we actually test a number of really prominent sites,

10:19 so Google, Facebook, Twitter, YouTube, and Amazon. And the reason we do that is because those sites are highly available, meaning they're likely not to go down. They're on networks where that are very well-managed, so, and they are diverse, meaning that some of them may use Amazon, but they're, and some of them use Google Cloud, some of them may have their own cloud, but at least when you're accessing those sites from a single location, you have a lot of probability that the network they live on will be actually a diverse network. Okay, so that's actually the sites that are used in the latency and loss and jitter test. For the DNS test, we have a number of different DNS servers defined here. The purpose of the DNS test is to actually test your own home DNS against

11:07 a variety of different public DNS servers. So these, these first three, okay, so Google, Quad9, Cloudflare, they're kind of here as the controls, meaning they're here; they will be testing this, or Net Probe will be testing these, and showing you the performance of these DNS servers versus whatever you put in here as your DNS server. Okay, so the idea here is that you put a DNS server in here that you use at home. Right, so if you need to, you can go into your command prompt and do an IP config, and then go look for your Ethernet adapter, oh, let's pull this down, let's look

11:53 at our Ethernet adapter, and my DNS, oh, I'll do an IP config /all, and there, there's DNS, and you can see my DNS servers defined in my local network are 10.10.50.0 and 10.10.1.1. You may have your own DNS servers provided here by your ISP, or you may have configured a public DNS of your own that you like to use. The idea is that you plug that in here and save it, so that you can actually map and see over time how well your own DNS server does versus sort of, you know, the big three DNS servers provided by the big companies. Okay, with the idea that, hey, maybe if you see that your server is lagging a little bit, you can actually go and switch to one of these better-performing

12:44 DNS servers that are better for your network. Okay, we also have some information about Redis. Yes, the password is "password" for Redis. This is all internal, meaning it's inside of Docker. Right, think of Docker like a virtual machine, like you have multiple little virtual machines spun up within Docker, which are the containers, and they all communicate with each other. So, Redis is one of the layers that transfers data between components. And yes, the password is "password," but it doesn't matter because it's not exposed to anything except for Docker. And then we have some information here about health score weights. So, if you remember the GUI for Net Probe, which I'll pull up, you see here, there's an internet quality score at the top. Now,

13:29 I mentioned this before, this metric is really just a made-up metric, okay. We take all the data we pull from packet loss, latency, jitter, and DNS response, we calculate those values together with various weights, we come up with a score. This score has no meaning outside your network. This is strictly so that you can keep track of the relative performance of your network, one day to another. Okay, so if you do want to go and tweak those values, you can go ahead. I do not support, um, or if you want to do it, do it, try it out, see how it goes. I can't offer you support if it breaks the application. These are the defaults that I've set. This is just to show you what those values mean. And then the last thing is just some more configuration for components that aren't really used. So, if you are modifying this file, a couple of areas I would recommend modifying:

14:20 number one, add your own DNS server here, under this line here, line 14; and, uh, and number two, if you do want to modify the sites, you can, just again, the disclaimer that if you use too many sites, if you break the line, it may break the application. And then number three, you're free to change these three kind of sample DNS servers, but just make sure it's three. The application is not really built to accommodate more, um, or feel free to go and modify the code to accommodate more. Okay, so let's get to actually launching the app now. So, let's launch the app. I'm in command prompt, in the folder with Net Probe Light, and I'm going to do one command to actually start up Net Probe and get it running, which is "Docker compose up." So, what this is doing, and

15:11 I mentioned Docker, think of Docker like, um, a virtual machine, uh, or virtual machine emulator, like VirtualBox, right, where, within Docker, to run Net Probe, there are multiple different containerized applications, or basically, mini VMs, that run applications that are all working together to produce the application Net Probe. And that's what all this output is, is all those different containers starting up, right, in the virtual environment. Now, uh, that is really it, right, you launch it, if you want to go back to the Docker window, you can actually see Net Probe Light is running in Docker. It tells you the actual, um, resource usage, which is actually very, very small, right. Net Probe only uses about 120 megabytes of memory,

15:57 which is nothing, and you, you can see here that you have five out of five containers running. And that's what I mentioned before, is that there's multiple containers that go into making Net Probe. All you really need to do now is from your own local machine, uh, let me pull up a window for that, open up your local machine, go to localhost:31, and you get Grafana. So, Grafana is the container that is the visualization piece of this. This is the one that shows you nice fancy graphs. When you first log into Grafana, it's "admin admin" for the username and password.

16:42 Once you've set that, it'll ask you for a new password. Right now, just for the sake of the demonstration, I'm just going to do "admin admin" again. You should set it to something within your home network that is more secure, and then you're into Grafana. Okay, um, from here, home dashboards, Net Probe, and there you are, you can see it's already starting to measure your internet quality score, your packet loss, your latency, your jitter, and your DNS response time. It's really that simple. Now, uh, within Docker, if you actually want to stop the application from running, uh, you can just come here and hit the stop button. Um, if you want to get it off your computer altogether, you just hit the trash button,

17:31 delete it, uh, and then delete your source code, go to that folder that the source code is in, and delete that. Um, that will delete all the historic data, as well as all the kind of temporary files that Docker may have created. Uh, but other than that, that's the footprint. Now, what I want to get into last is really just usage of Net Probe. How do I actually use the tool to monitor and troubleshoot the network? And for that, you can see I've got my localhost that I just launched, but I, I actually want to open up a version of Net Probe that's running on a different server in the same network, so you can see the IP address here is 10.10.50.0. Um, and this, the same exact code, it's just I've had this running for a lot longer and collecting data, to allow me to actually show you how Net Probe works. So, you run Net Probe, you let it run on the machine, you leave that

18:16 machine on 24/7, um, it's going to collect data. If you restart the application or the computer, Net Probe by default is configured to restart when Docker starts, so just make sure that Docker starts at your startup, which it should, and then the data will be preserved. So, even if you restart, you can see here I had a little restart of the machine. If I zoom in here, um, it, uh, the data is preserved, and it'll just continue monitoring data after that. So, let's talk about how to actually use Net Probe to monitor your internet traffic. The way that this works, if you've never used Grafana before, Grafana offers dashboards which have these various metrics that you can go through. You can actually change the time window up here, so let's say I want to do 24 hours versus 2 days versus 7 days. It'll really let you go up to the maximum amount of time that

19:05 you've got collected on disk. And really, I've run Net Probe on other machines for upwards of a year, um, uh, in the beta version without any issues. So, um, feel free to continue to let it run, and it'll keep collecting data. There's no issue. Um, let's start from the bottom up, uh, to understand how to use the data. Now, number one, DNS response time. As I mentioned in the previous section, what this is doing is doing DNS queries to a bunch of different websites. Actually, the five different websites that I have mentioned, uh, in the, uh, configuration file. So, this is actually going to be, uh, these five sites. Okay, so these sites are used for the ping test but also for the DNS test. It resolves the name of each; it averages the speed, uh, at which those resolvers

19:52 get you a response. So, like, a time it takes to actually get a DNS response from Cloudflare, from Google, from my home DNS, whatever you set this to, and also Quad9, and then displays the data here, along with your average response time for your own DNS server. So, this response time here is from whatever you configure in the environment file as your DNS server. Okay, um, with the idea that this allows you to keep track of, over time, how these servers are performing relative to one another, because DNS could be a major component in network slowness. If you have a DNS server that is consistently slow, every single one of the transactions you do on the internet, or any of the devices in your network, are going to be slow by a certain percentage, right? Could be a lot, could be a little. So, this helps you keep track of that. And what you could do is actually click on individual servers to display their performance individually. Right, and you can

20:45 do, you can get a relative sense of which one is performing the best over time. So, right now, it looks like, if anything, Google DNS and Cloudflare are, are the winners, or at least from a visual inspection, but I can actually go back and forth in time, compare different time periods, to see which DNS servers are actually best here. But that's how the DNS module works. Um, let's go in mixed order to packet loss. So, packet loss, what packet loss is doing is sending those ICMP pings out, uh, it's actually sending quite a number of pings, not just like five pings at a time. Net Probe is sending, uh, 50 pings every 30 seconds to each of these websites, um, and it

21:31 is recording back the, one, the packet loss. So, how many of those packets were lost in transit, going to and from those websites. Packet loss on its own is not a huge issue, if it is not chronic, and if it's happening once in a while. Okay, um, because there's always going to be some percentage of traffic that's lost on the public internet. You run into problems when packet loss is chronic, like you're seeing packet loss all day, every day, um, or if packet loss is, uh, you know, it's quiet, quiet, quiet, and all of a sudden you have like a big period of packet loss for a day or so, and then it goes away. So, that's what this is measuring, and this is what's showing you. Um, I set a threshold at 4% for a reason, and that's because with TCP, specifically, so TCP-based protocols, when you hit 4% packet loss, uh, TCP degrades to a point of almost unusability, right.

22:23 Um, it has to do with TCP congestion control and the algorithms that are built into TCP, but really, if you're a network engineer, by rule of thumb, you should have as close to zero packet loss as you can, but once you hit past this point, and if it's sustained past 4%, you have a serious problem where, um, you need to get that inspected, whether it's bad cabling, whether there's something else going on in your modem, um, that's going to hamper all connectivity in and out of your network. Now, I'll see, see if I can find an example of that. We had some extreme packet loss in this section. With Grafana, it's quite nice, because if you want to zoom into a little bit of data on our time window, you can actually just click and drag, and that exposes this. Now, take a look at this. This is a short period of time where I had extreme packet loss, up to about 7%,

23:15 and you can see that had a very negative effect on my overall internet quality score, obviously, because this is a calculation based on packet loss, as well as the other metrics. Um, but that also had an effect on latency, right, it also had an effect on my jitter, a little bit spiked up, and it looks like DNS response times also went higher than they should. Okay, so this is how you kind of drill in to different time periods, but how it makes it easy to, uh, take a look at data, zoom into some pieces of data, and take a look at instances where you have things like packet loss. Now, in this case, I'm not as worried, because this was a short-term event, but this kind of thing, um, if it happens on a regular basis, like, you would certainly feel it. Okay,

24:01 so let's switch back to a 24-hour view. Uh, next is latency. So, the same ping test that we're running to measure packet loss, we are measuring latency, um, to various different websites. Now, you may have the very valid question, well, how does it make sense that we have a one latency figure, because Facebook could be 50 milliseconds away, and Google could be 20 milliseconds away, and Yahoo, or the other website we're testing, could be 100 milliseconds away. What does this metric mean? And that's a very good point. Um, the idea here is that we measure the latency to these various sites and average them out, um, and what we're keeping track of is whether or

24:46 not overall that latency increases substantially, because something that can happen within an ISP network is that because of things like buffering, like over subscribing links, like they don't have enough capacity in the ISP network, you could have latency increases that affect traffic to all these websites that we're, we're probing. And this is what this is trying to keep track of, it's for you to understand what the baseline latency is to all these different sites, and if you do see increases over and above your baseline, then you may have an issue that's not due to one of those sites, it may be due to your actual ISP. So, if I look at the 7-day view, you can see that I've actually had, let's see here, there's some times where the latency is actually quite stable, and then we

25:34 have these spikes, um, and then we have this day where I had that really poor performance, and then I actually had overall latency go up and down over and above the average, and then come back down to a very settled place. Um, now, I'm very lucky because my ISP is actually relatively stable, but you may see some weird statistics and ups and downs in your graph that you may want to investigate and dig into a little bit. But that's effectively what latency is measuring. Okay, jitter is the rate of change of latency, so this one is particularly important if you're into real-time voice and video, or if you're into gaming because, um, if you're, if someone says you lag, right, you're playing a game, you lag, uh, it could be two things, right. One, it could be that

26:22 you're latency to and from the server in question, so like, say it's a video game server, or like a, uh, a server for online gaming, you could have high latency to that server, meaning every one of your actions takes a long time to get to and from that server. Uh, that's one thing. But it could also be that your jitter is quite high, meaning your latency, so this value here, is going up and down a lot, it's not staying at a constant 24.2, it's actually going up and down very rapidly because what that's going to do is change how often your, the interval between your responses to that server. So, say, say you're, you're playing like an FPS, um, and you're, every time you click, every time you move, every time you make an action, that is a packet to and from the server,

27:07 or a message to and from the server, and if those actions are happening at different intervals, like at one point it's 20 milliseconds, another point it's 30 milliseconds, and 20 milliseconds, that's high jitter. And what that results in is a very unstable performance of real-time. So, if you're into, again, if you have real-time video, real-time audio, um, or you are into gaming, you got to keep track of jitter because that's really important. And if you see huge spikes in jitter, or like periods where you have high jitter, um, that could be a serious problem. So, all those metrics, as I mentioned, go into the internet quality score, which you can keep track of over time, um, and there are different scenarios that we can look at that, uh, for different problems that you may have on the network. But really, I just want to leave it there because I

27:53 think the main thing for this is to set it up, let it run your machine for a period of time, and then observe, keep track of it. What I would also ask you to do is, once you set it up and run on your ISP for a period of time, come into the Discord channel and share your results because I'd really be interested in seeing the relative performance of everyone's individual network because I think it's really interesting that we expose this information for each other to see, to see what kind of experience we're having. The relative score is going to be, you know, not really relative to everyone else, but I'd be interested in seeing how much packet loss people are experiencing, and in an average 7-day period, how much latency, right, how much jitter, and what the DNS response is like from their home ISP. That's it for this video. If you enjoyed it,

28:39 please like and subscribe, and if you have any questions on the topic we've covered, please join the Discord server, where we talk all things network. Until next time.