An Exercise in Frustration: Fine Tuning the Web Filter in Sophos UTM

Everything up til know regarding the Web Filter has been a cake walk by comparison. I’m not saying it’s been easy, but fine tuning the web filter, adding exceptions and the like is going to be the hardest part of this by far.

I’m going to cover some of the basic services that you’ll need here, and maybe a bit of how to troubleshoot to identify future issue. But this part will cover the part that has caused me the most frustration and the most time.

Unfortunately, not every website, platform or app supports the Web Filter, even when you have the CA Certificate installed. In fact, some programs store the CA Certificate store outside of the OS, making it problematic.

As I’ve said before, if you’re not willing to deal with the frustration and troubleshooting, turn back now. Go back to the Web Filtering section and on the HTTPS tab, set it back to “URL Filtering”. Or just turn off the web filtering altogether.

First, lets set up some new exceptions. To do so, open the Web Protection section, and click on the “Filter Options” entry. This should take you to the Exceptions tab by default.

This is where you’re going to be spending a lot of time, so get used to it!

Filter Options - Exceptions


For most of these excepts, we are going to want to enable the same options. This will allow most everything to work properly.

Adding Exceptions for Common Services

To add a rule, click on the “+ New Exception List….” to add a rule. The options you want to check is “Authentication”, “Antivirus”, “Extension Blocking”, “MIME type blocking”, “URL Filter”, “Content Removal”, “SSL Scanning”, “Certificate Trust Check”, “Certificate Date Check”, and “Do not display Download/Scan programs page”.

If you want, you can edit the rules afterwards and un-check options, that way, you’re only exempting the necessary parts, and filtering everything else that you can.

Each rule should look like this:

Filter Options - New Exceptions


When you add a rule, select the “Matching these URLs” option in the “For all requests” option at the bottom of the “Add Exception list” section. You can use normal URLs here, or (ideally) “RegEx” strings to filter out what URLs you want.

However, the easy way to add batch lists is to click on the icon to the right of the plus sign and select the “Import” option. This will bring up a text box that will allow you to copy and paste multiple entries at one time.

Filter Options - New Exceptions URL

These is not a comprehensive list by any means, but it is a fairly thorough and tested one. As you can see here, I’ve put a lot of time into testing this out. A … LOT.


This rule covers a bunch of sites that are used … a lot of places.

Add the following entries, one for each line:


The last entry could be problematic, as it allows any direct connection to an IP address via HTTP/HTTPS. However, Twitter, Facebook, Netflix, Google, TwitchTV, Skype, WLMessenger, Nintendo, and Playstation definitely use direct IP addresses for content, so it’s a necessary evil (trust me on this one… if you don’t turn the rule off and see how useful the internet becomes).


Add the following entires (one for each line):


The first rule gets about anything at netflix (,,, etc). The rest get other various netflix URLs.





Trillian and various IM/Chat services






Without this, ChatWing’s messages don’t seem to update properly (only an issue on site that I’ve seen).

Cloud Services


StableBit DrivePool/Scanner updates




Google Music








Between the first rule and the “CDN” rule, you need this to be able to download the installer bits, or they will crash during install.



Some users have reported issues with the Microsoft services, these should help fix the issue. The last two are for Microsoft’s Cortana service.

Microsoft XBOX





The first two (and the CDN  rule above) are the only ones needed for stock Minecraft. The rest are for various modded versions. But I feel it’s better to include them.

Nintendo (WiiU/3DS)*




PayPal Here


PlayStation Network






Raxco PerfectDisk




Steam Gaming




WordPress Services







Any service with an Asterisk does require the “CDN” group as well. The “CDN” group is basically a group of sites that are used by many services.

Additionally, I do plan on keeping this list up to date and adding additional information, when possible. But as I don’t own every device or software package on the market, I can’t test everything.

Though, feel free to donate money or devices to me to help with this.

A Brief Explaination

You may see a lot of things above and may be confused. So let me break this down (a crash course on RegEx, “Regular Expressions”).

The “^https?://” part indicates that the URL must start with http:// or https:// (the “?” behind the “s” means that it’s optional, it can be there but it doesn’t have to be). This makes it so you don’t have to input two sets of URLs.

The “([A-Za-z0-9.-]*.)?” part before the main domain means that it can be the main domain (such as “”) or any combination of letters, numbers, “.”‘s and dashs. This means it will detect just about any sub-domain of the main domain (such as “”, or “”). This makes it so you don’t have to input every URL in question.

Also, since this is “RegEx” notation, every period (“.”) in the URL needs to be “escaped”, so that it knows it’s a period. So you should replace them with “.” to make sure it works right.

You should add each site or service as a different rule, that way you can turn them on or off easily, for testing.

Ad Blocking

This namely comes from here:

I’ve used this, but it may be over zealous in blocking things.

Under “Web Protection”, click on the “Application Control” link. Then open the “Application Control” tab. Create a new rule, and click on the folder icon next to “Control these Applications”.

This will bring up a window with a large list of websites. In the “Categories” section, find “Web Services” and only select that. In the filter/search bar (next to the “Application” column header), try input “Ads”, “track” and then “analy”. This will filter the list and make it easier to filter out different websites/applications.

Filter Options - Application Control


This will help filter out out ads, tracking sites, and analytic sites, if you are so inclined. Click on “Save”, give the rule a name (such as “AdBlock”) and save the rule.

This will help speed up browsing and reduce ads, but we’re not done. Time to head back to the “Filter Options”. Once you do, open the “Websites” list.

Click on the “+ New Site…” button. This will bring up a popup window, with a text box. I’ve attached a text file, download this, and add the contents to the “URLs, domains, IP addresses or CIDR ranges” box.
Sophos WebAds
I have removed some analytics/tracking sites, such as twitters and Google’s since many webmasters may use these for feedback.

Check/Tick the “Include sub-domains” entry, set the “Category” to Web Ads (this should be blocked if you are blocking the “Suspicious” category, and set the “Reputation” to “Malicious”. Hit “Save” to save the list.

Filter Options - Websites

Once those are added, we want to add a couple of more. Click on the “New Rule” button again. This time add the following text:

Do not check the “Include sub-domains” option this time, but do change the Categories to “Web Ads” again, and set the reputation to “Malicious”. Hit “Save”, and we’re mostly done here!

This should block a lot of ads now. However, you may notice Sophos screens popping up in your websites now. This is fine, and just an artifact of the blocking process. The reason it’s fine? The server is local (it’s your router), and the connection to it should be incredibly fast.

You can fine tune this to include additional sites, such as, if you want. Or if you run into sites that are blocked, add them here, and add them to a better category and set the reputation to “Trusted” or “Neutral”.


I won’t lie. A lot of what you’ll really be doing in the web filtering is troubleshooting what’s going on. If you hate sleuthing and digging into logs, then again, turn back and turn off the “Decrypt and scan” option and just do the URL filtering.

The first thing here, is that you had better get used to reading logs. It’s going to be a key to identifying the URLs in use and fixing the issues.
You can always ask me, but I may not be able to identify and fix the issue for you.

At the top of the page, on the right side are a bunch of icons. The one we want is the “clipboard”. This opens the “live log” for the current section. So as long as you’re in any of the Web Filtering pages, then this clipboard will open a new window with the live log for the web filter.

Live Log - Web Filter


This is a very detailed log. Each line contains info about the requests, including the method (POST/GET/CONNECT”, the action, the source and destination IP addresses, the user account (if you enabled that), the status of the request, the filters, the size of the response, the URL, referrer and … more. Here is an example of ONE of the log:

2015:03:19-11:12:51 sophos httpproxy[6503]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="POST" srcip="" dstip="" user="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="156" request="0xdcc8800" url="" referer="" error="" authtime="0" dnstime="1" cattime="309" avscantime="11258" fullreqtime="221600" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36" exceptions="" country="United States" reputation="neutral" category="148" reputation="neutral" categoryname="Shareware/Freeware" content-type="text/plain"

To make matters worse, the logs auto-scroll, and because of the amount of data… it may rapidly scroll.

Live Log - Web Filter logs


Disabling auto-scrolling may be a good idea when you’re looking through the logs. Also using the search feature of your browser may be very helpful.

The first thing to look for is the source IP address. Look for the IP address of the device having issues. Then look at the URLs. You may see the URL for the services you’re using. If you do, then add these to the exceptions.

If you’re seeing a bunch of URLs using the same domain with a bunch of different sub-domains, then use the “^https?://([A-Za-z0-9.-].)?” text before the main part of the domain. Eg: “^https?://([A-Za-z0-9.-].)?” . This will catch ALL of the possible subdomains for “”. If you’re only seeing a few being used, then grab those specifically. Just remember to “escape” the periods in them (use “.” instead of “.” for the URL), and to start off the URL with “^”.

However, be on the lookout of “weird” URLs as well. Sometimes, web services use “CDNs” to deliver their content, or use weird backend servers. This can cause issues with identifying services.

So the best advice is to keep on trying to add new rules/URLs and retrying until you’re able to access the service. Then try to delete URLs until you are sure you only have the bare minimum of what you need.


In the Exception list, there are other options other than just the URL. You can actually specify “And/or” values as well. For instance, you can specify downloads are not to be scanned from a certain domain, but only for a specific computer.

To do this, add the URL, and then underneath that, there is another section. Set the first drop down list to “and” and the second to “Coming from these Networks”. This gives you more control over what gets blocked and where. This way, you can block minecraft updates for all but the family computer and xbox, if you want.

Filter Options - Exceptions - Advanced

Exceptions Don’t Work!!!

However, with everything that you can do and check and exclude… sometimes no matter what you do, it may not be enough.

In this case, there is a “nuclear option”. If you open the “Misc” tab on the “Filtering Options” page, there is a “Transparent mode skiplist”. This list allows you to completely bypass a website, or exclude a host from being scanned.

Filter Options - Misc - Skiplist


In the “source hosts/nets” section, you can specify the PCs/devices on your network that you want to bypass the web filter.

Additionally, if you need to turn off the web filter temporarily for testing, I would recommend adding the “Any” “device” here, as it effectively allows all devices to bypass the web filter. This is better IMO, because turning the filter on and off temporarily disconnects you from everything. This does not.

However, make sure the “Allow HTTP/S traffic for listed hosts/nets” option is enabled, or you won’t be able to access the internet from these devices.


And That’s All Folks!

After this, there really isn’t much else of anything that we really need to do. Everything should be working for the most part, and there should be enough info here to help you troubleshoot any issues you have.

If you’re still having issues, then please let me know in the comments or via the contact form. I’ll try to help you as best I can.

Author: Drashna Jael're

Drashna Jael're

76 thoughts on “An Exercise in Frustration: Fine Tuning the Web Filter in Sophos UTM”

  1. I found I could copy the lines from your website and paste directly into Target Domains. This saved me hours of work.

  2. #2 Son complained that he could not access twitter. I had no problem. Turns out I am accessing via my app (Tweetium), he was just trying to get to twitter via selected goolgled tweet IP’s.

    I have added “^https?://([A-Za-z0-9.-]*\.)twitter\.com/ to the filter exceptions which has solved his problem. Started with 10 exception that came with Sophos and am up to 17. How long will the list get?!

  3. you can also paste multiple patterns using the ‘import’ menu instead of one line at a time, and don’t forget ‘clone’ to inherit the same initial settings

    1. Neil,
      I’m not sure what you mean by “import” menu.

      I know you can add multiple URLs for the “Website” section, and I should clarify that, but otherwise, i’m not sure what you mean.


  4. @drashna,

    I wanted to pop in and say thanks for putting this together, it is greatly appreciated! As a recent convert from Untangle having a concise well-written guide to help me sort through things helped a ton!

    Regarding the “Import” comment above from Neal, I just looked closer at the “Add Exception List” dialog and on the end of the “Target Domains” table there is a plus sign “+” followed by a drop-down menu graphic. If you click that you will see the following:

    – Import
    – Export
    – Empty

    Selecting “Import” will allow you to paste your rules above in and it will create separate entries, which massively simplifies things! The same goes with “Export”

    Thanks again @Neal !

  5. I have set up UTM 9 and enabled the web proxy restrictions. I have enabled https filtering and imported the CA to an android device successfully. I have selected streaming media as one of the blocked categories. I’ve even added and to the websites tab and marked both as streaming media under the category drop down.

    Then just in case the device was going out the other gateway in the home I enabled the DHCP server in the UTM and validated the android was getting the correct IP info from it. I have ensured the wireless router is pointing to the UTM gateway as well.

    The logs show the traffic still being allowed…

    2015:06:09-11:07:00 sophosutm httpproxy[5397]: id=”0001″ severity=”info” sys=”SecureWeb” sub=”http” name=”http access” action=”pass” method=”CONNECT” srcip=”″ dstip=”″ user=”” ad_domain=”” statuscode=”200″ cached=”0″ profile=”REF_DefaultHTTPProfile (Default Web Filter Profile)” filteraction=”REF_DefaultHTTPCFFAction (Default content filter action)” size=”11489″ request=”0xe4545000″ url=”https://r9—” referer=”” error=”” authtime=”0″ dnstime=”28572″ cattime=”0″ avscantime=”0″ fullreqtime=”233076″ device=”0″ auth=”0″ ua=”” exceptions=”av,content,url,ssl,certcheck,certdate,mime,cache,fileextension,size”

    I am stumped… Thoughts?

    1. Are you trying to block youtube and the like?
      If so, open the Filter Options section, and find “YouTube”, and turn it off (click on the green thing next to the rule’s name).

      If that’s not what you’re looking for, then let me know what you’re trying to do explicitly (and feel free to email me at “” directly).

  6. This guide is great, exactly the kind of thing I have been looking for to cut down the time to learn regular expressions. I have had pretty good luck trying to block or allow specific domain, but I will definitely be implementing some of this filters.
    I do have a question, how would you go about adding filters to look for specific strings in a search, for example when using Google the url generated after entering a search term will contain the terms in the search. If i wanted to block searches for “cars” the url that results is “” or something simliar. How would you go about block the urls for other unsavory or explicit web searches?

    1. Check out the “safe search” filter option. This should allow you to forcible enable the safe search filter for the systems, essentially negating the need for that.

  7. Thank you for your posts on Sophos UTM. I have still stuck on getting Netflix to work on Android cell phones and Apple iPads.

    1. Steve, I’m glad that you like the guides and have found them very helpful (I’m assuming).

      Sophos does have manuals… but they’re all text based, and assume that you have experience with administrating a network. I’ve written them assuming that you don’t, so that’s the major difference! (also, mine may be easier to find)

    1. Actually, a singular rule will get all of this (and I’ve added that):

      This is a RegEx string, and gets http/https for * or *, with just the one string. It’s cleaner, IMO. But it may allow for additional URLs.

      But, thank you for posting that!

        1. No, you didn’t miss it actually. I only just added it.

          However, I’ve become somewhat familar with “RegEx” which is SUPER powerful for searching and matching content in text strings.

          Which means that instead of having ten different search strings, you can use just the one and get everything.

          To explain a bit:

          The ^https?:// part means that the url can start with http:// or https:// (but must start with these).

          the ([A-Za-z0-9.-]*\.)? part means that the URL doesn’t need a sub domain (such as www), but it will get ANY subdomain here.

          The hulu(im)?\.com/ part means that it grabs or

  8. I recently experienced issues with my Sophos UTM 9.315-02 where iOS devices were exhibiting slow, painful web browsing performance. After much troubleshooting, it turned out to be due to the Adblock definition I created above following the Application Control guide. Have you experienced this? Seems like either certain legitimate sites do quite a bit of tracking OR I had too many things selected.

  9. Thank you so much! Install and basic configuration was breeze, but getting some services going was a real pita.

  10. Hi,

    Have you any solutions for icloud? My UTM blocks all incoming and outgoing and traffic, its driving me nuts as I cannot find a solution!

    Any help would be appreciated! Thanks!

  11. Hi, I found this very useful.. I am trying to do a filter based on users. So if a username starts with let’s say 60 then only show them users. I can’t figure out how to do this. Thanks

  12. Thanks so much for the level of detail. Being new to Sophos coming from another UTM vendor, this saved me loads of time. Not to mention made my wife happy that Netflix was working again for her and the grandkids. Thanks again for all the work putting this together.

  13. You, my good man, are my hero. Thank you, thank you, thank you for publishing this. I know it took you countless hours, and you are saving me a few good weekends of work, and I would likely just turnoff HTTPS inspection at the end of it all. Bless you!

  14. Thanks so much for this guide it has been do useful in setting things up. I do have a question however, a few weeks ago Facebook changed something in an android app update and now no images will load. How can I detect what they have changed? I can’t see anything in the logs that is being blocked? Am I looking in the right place or do I need a third party tool?

    Thanks in advance,


    1. That’s odd that Facebook would do that.

      Make sure that you’re using the Facebook rules I’ve posted. This should allow both the main facebook site through without messing with anything, and their CDN sites as well ( and

      As for figuring out what is being blocked, I’ve already explained exactly how to do that in the “Troubleshooting” section at the bottom. This section goes over exactly what to do and look for to fix the issue.

      And unfortunately, I’m not able to confirm the issue you’re seeing.

  15. Hi, great article and much better than Sophos.
    Do you have any tutorials on how to use “Bypass Users” option please?

    Trying to set up a small free WiFi, but would like to bypass certain downloads and sites using the web browser. I thought it might bring up a screen like… “this site is blocked etc…” Enter name and password to access.. I have added an admin user to the device, but cant get the screen to unblock..

    I am sure I have seen this some where?

    Thanks in advance


    1. Well, bypassing specific users is more complicated. First, you need to set up the filter to use specific user accounts, which is a bit more difficult. And then you can set up different policies for different users.

      Barring that, you can add the websites that you don’t want downloads scanned to the exceptions list. Just create a new entry, uncheck the “block by download size”, “antivirus”, “extension blocking”, “MIME type blocking” and “do not display download/scan progress page” options. Add the URL as per normal, and it should prevent Sophos from scanning the downloads from the site in question.

  16. I have been reviewing my own rules and revisted this page. I am wondering if there should be a ‘standard’ in these or there were intentional differences in the coding – sometimes there is a following / and sometimes no, for example, and sometimes a ? prior to the major domain and sometimes not. for example, ^https?://([A-Za-z0-9.-]*\.)?githubusercontent\.com versus ^https?://([A-Za-z0-9.-]*\.)akamaihd\.net
    I think the omission of the ? will prevent it from matching just plain Is that the intent?

    1. The ? means that the previous “character” is optional. For instance, “https?” means that it can be http:// or https://. So removing the ? after the parentheses would mean that it *has* to have a subdomain.

      As for the trailing /, that’s a formatting issues that I could clean up.

      And as for why some rules include the subdomain stuff, and some don’t? Well, I want to be as specific as possible, but entering 10 different, similar URLs is a PITA. Easier to use the subdomain stuff there. Especially if the service changes the exact URL or uses a semi-random one.

      1. really my question is should I include the ? and the trailing / in ALL my entries or is there method to the madness…

          1. a thought about the trailing /, it seems like a security thing, avoiding subdomains that mimic the target. for example, if you want to make an exception for and leave out the /, you would also have an exception for or . now I am starting to lose sleep over this….

          2. I could probably clean then up and test them more thoroughly for security.

            Not sure if I’m going to right now. Still deciding which to use.

  17. Hi, exclent post! Maybe you can help me about webfilter on Sophos 9.3.
    My Sophos is joined on domain controller, everything is ok.
    I need to configure 2 groups, i created 2 groups on AD like Web_full and Web_limited right.
    Need it to be authet, i created new web filter profiles, then add my network and transparent mode with AD SSO, i created filter actions like web_full, on AD i add user administrator on web_full and add other user on web_limited right. I’m so confused about the order. When i did some test, the i block everything on web_limited, but that user in, can browser on the web. How can i make this work?
    I appreciate your help!

    1. Unfortunately, I can’t help you here much, as I’ve never messed with the user groups.

      I had planned on looking into it, but … life came up and Sophos is changing the UI completely. So I decided to wait on it. Sorry.

    1. Chris, thanks for this update. At our location we do *not* want to allow all of but do want to allow Blizzard updates so if you have any additional servers that should be allowed I would appreciate hearing of them!

      1. Nope. I’ve sat and watched (recently actually). Blizzard is pulling from directly.

        Aside from finding the specific URL path it’s using and ONLY allow these, the other option would be to allow the updates to occur on one computer (and isolated system), copy the files onto the main network and then push them out from there (such as for group policy).

  18. ^https?://([A-Za-z0-9.-]*\.)facebook.\com/



    seems need to be changed…

    should be “\.com/” at the end?

  19. and another thing…
    in the CDN list…, especially with: “^https?://[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*”

    this seems to defeat the filtering/firewall altogether 🙁
    as I interpret it like “all addresses (or domain)” will have access.

    when I’m trying to debug/see the logs on my sophos [thanks to your guides here], I really do see some domains without names but all numbers only, this maybe what youre trying to do for this particular regex.

    as far as I understand with the so called DNS, it will be ultimately be translated to pure numbers like from for example or whatever your website(domain) is.

    or…, I’m wrong altogether???

  20. Hi,

    Thanks a lot for taking the time to do this! I#m still trying to get plex up and running. I’ve went the Nuclear route and allowed traffic through without being scanned but I’m still get “target service not allowed”

    Have yuo encountered any problems with Plex?

    1. I don’t actually use Plex, so I’m not 100% sure why it would be doing that.

      Have you tried adding the system to the skip list? If that helps, then it’s requesting something that is being filtered out still.

      Though, it sounds more like one of the ports that Plex wants to use is not opened properly.

  21. I had to add this for PlayStation Store:

  22. This site has been extremely useful but I was wondering if you could clear up a question I had – both here and on Sophos web site there are patterns that end with / (forward slash) and some that do not. Would you mind elaborating for us non-programmers why the variance? Is it needed for some sites and if so, why? Thank you in advance!

    1. They should all end with a forward slash, honestly. If they don’t, it’s because somebody is being lazy.

      That way, it makes sure that the .com or whatever is the end of the URL, so you couldn’t use something like as well. It shouldn’t happen, but just in case.

      1. Thanks for the unexpectedly prompt clarification! So does this mean that both and would apply to ? Or in other words the / won’t affect a domain by itself?

  23. Thank you for the time and effort you took to construct this. 2 years later I am finding this very useful.
    When I constructed the exceptions from your list only needed to check “SSL Scanning”. All other options remained un-ticked and I’ve had no issues so far.

    I also noticed I could not sign in to my work’s Skype for Business or Citrix Receiver (Storefront) from home so I added the following:

Leave a Reply