@webknjaz's ramblings

Get Off My LAN: Banishing a Google Home Smart Speaker with OpenWRT by Mistake

2025-11-27T01:12:00Z

A Story of Exile

It's not DNS
There's no way it's DNS
~~It was DNS~~ but not really, no

Sometimes you don't mean to exile your smart devices to network purgatory, but your favorite router operating system gives you the power to do it anyway. Here's how I accidentally banished all my Google Home speakers from the internet with a seemingly innocent DHCP configuration change.

The Unwitting Exile Begins

My old Google Home speakers stopped working one day. They'd claim no internet connectivity on any voice command — as if they'd been cast out from the network, unable to reach the digital world beyond. Factory resets didn't help. They'd get stuck during initial setup, insisting they couldn't connect to Wi-Fi.

The maddening part? Setting up a mobile hotspot on my phone and pointing the speakers to it instead of the home network worked just fine. So the hardware wasn't the culprit. And this also meant that Google hadn't deprecated them. They just couldn't live on my LAN anymore.

Signs of the Banished

Initially, I blamed a recent OpenWRT update I installed onto my access points. The timing seemed suspicious. And the fact that I use my own ImageBuilder-based immutable deployment process wasn't making the guessing game any easier. But after multiple frustrating troubleshooting attempts, I noticed something telling in the AP's web interface: the Wireless page listed the device with its MAC address, but no corresponding IP or hostname — like a ghost, present but not truly there.

Yet I had IPs statically assigned in dnsmasq on the main router. Looking up the device's MAC address gave me the expected IP, and pinging that IP worked fine.

Why would my Google Home speaker think it's air-gapped when I could reach it over LAN? It was connected yet convinced it wasn't — the perfect network gaslight.

Tracking the Exile Order

The AP's logs showed no problems. The device connected to 5 GHz, didn't like it, disconnected gracefully, and hopped to 2.4 GHz — all normal behavior for a device trying to find its place.

Time to intercept the DHCP conversation and see if the speaker was getting the configuration responses correctly. Here's what I grabbed with tcpdump, having been SSHed into the nearest AP:

$ tcpdump -i br-lan '(udp port 67 or port 68 or port 546 or port 547) and ether host f4:f5:de:ad:be:ef' -vvv
01:13:36.373309 IP (tos 0x0, ttl 64, id 13202, offset 0, flags [none], proto UDP (17), length 352)
    Google-Home.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from f4:f5:de:ad:be:ef (oui Unknown), length 324, xid 0xc06f922f, Flags [none] (0x0000)
     Client-Ethernet-Address f4:f5:de:ad:be:ef (oui Unknown)
     Vendor-rfc1048 Extensions
       Magic Cookie 0x63825363
       DHCP-Message (53), length 1: Request
       Requested-IP (50), length 4: Google-Home
       MSZ (57), length 2: 1500
       Vendor-Class (60), length 41: "dhcpcd-6.8.2:Linux-3.8.13+:armv7l:Marvell"
       Hostname (12), length 11: "Google-Home"
       Unknown (145), length 1: 1
       Parameter-Request (55), length 9:
         Subnet-Mask (1), Static-Route (33), Default-Gateway (3), Domain-Name-Server (6)
         Domain-Name (15), BR (28), Lease-Time (51), RN (58)
         RB (59)
       END (255), length 0
01:13:36.377212 IP (tos 0xc0, ttl 64, id 50324, offset 0, flags [none], proto UDP (17), length 373)
    turris-omnia-gw.67 > Google-Home.68: [udp sum ok] BOOTP/DHCP, Reply, length 345, xid 0xc06f922f, Flags [none] (0x0000)
     Your-IP Google-Home
     Server-IP turris-omnia-gw
     Client-Ethernet-Address f4:f5:de:ad:be:ef (oui Unknown)
     Vendor-rfc1048 Extensions
       Magic Cookie 0x63825363
       DHCP-Message (53), length 1: ACK
       Server-ID (54), length 4: turris-omnia-gw
       Lease-Time (51), length 4: 172800
       RN (58), length 4: 86400
       RB (59), length 4: 151200
       Subnet-Mask (1), length 4: 255.255.255.0
       BR (28), length 4: 192.168.1.255
       Default-Gateway (3), length 4: turris-omnia-gw
       Domain-Name-Server (6), length 4: turris-omnia-gw
       Domain-Name (15), length 9: "home.lan"
       Classless-Static-Route-Microsoft (249), length 9: (10.60.0.1/32:turris-omnia-gw)
       Classless-Static-Route (121), length 9: (10.60.0.1/32:turris-omnia-gw)
       END (255), length 0
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

There's our smoking gun: extra DHCP options 121 and 249! I started remembering that I'd been experimenting with pushing explicit routes to my ISP's internal network nodes via DHCP — unknowingly drafting the exile papers for my smart speakers.

False Trails in the Wilderness

I opened LuCI's network interfaces page of the router that runs a fork of OpenWRT and found additional option entries in the advanced DHCP settings. Deleted them, restarted dnsmasq, and... nothing happened. Nothing! The exile continued.

The 121 and 249 options were still being sent. The banishment order was coming from somewhere else.

The Hidden Decree

Checking the UCI configuration revealed the true source of exile:

root@omnia:~# uci show | grep 249
dhcp.lan.dhcp_option_force='121,10.60.0.1/32,192.168.1.1' '249,10.60.0.1/32,192.168.1.1'

I'd configured forced DHCP options, apparently — a setting not exposed in LuCI, only accessible via uci commands. These were the real culprit, hidden from the web interface.

Revoking the exile was simple:

root@omnia:~# uci del dhcp.lan.dhcp_option_force
root@omnia:~# uci commit dhcp
root@omnia:~# /etc/init.d/dnsmasq restart

Understanding the Banishment

Why did this shut out my Google Homes? RFC 3442 holds the answer under the DHCP Client Behavior section:

"If the DHCP server returns both a Classless Static Routes option and a Router option, the DHCP client MUST ignore the Router option."

DHCP options 121 and 249 don't just append routes — they become the only routes the client would use if it supports them. By providing just a path to 10.60.0.1/32 without a default route (0.0.0.0/0), I'd essentially told my Google Homes: "You can only talk to this one host. The rest of the internet doesn't exist for you."

It was network isolation through misconfiguration — an accidental digital exile where the devices were physically connected but logically banished from the wider network.

Granting Amnesty

The proper fix maintains the specific route while granting passage back to the internet:

root@omnia:~# uci add_list dhcp.lan.dhcp_option_force=121,10.60.0.1/32,192.168.1.1
root@omnia:~# uci add_list dhcp.lan.dhcp_option_force=249,10.60.0.1/32,192.168.1.1
root@omnia:~# uci add_list dhcp.lan.dhcp_option_force=121,0.0.0.0/0,192.168.1.1
root@omnia:~# uci add_list dhcp.lan.dhcp_option_force=249,0.0.0.0/0,192.168.1.1
root@omnia:~# uci commit dhcp
root@omnia:~# /etc/init.d/dnsmasq restart

The exile was lifted. My Google Homes could rejoin the digital society ~~and get back to spying on me 🤪~~.

Lessons from the Exile

OpenWRT gives you the power to banish devices from your network in subtle ways. DHCP options 121/249 don't supplement routing tables — they replace them entirely, making them perfect tools for accidental exile. LuCI won't always show you the full picture; uci show reveals the hidden decrees. And while many devices ignored my misconfigured exile order, Google Home's RFC-compliant DHCP client (dhcpcd-6.8.2) dutifully accepted its banishment.

When your smart devices claim they're offline while clearly connected, check if you've accidentally exiled them. Sometimes the most effective network isolation is the one you didn't mean to create.

Well, it wasn't DNS after all — just a routing misconfiguration that prevented reaching it. Though the haiku's spirit lives on: when troubleshooting network issues, DNS is always a suspect, even when it's innocent.

ansible-galaxy CLI ❤️ resolvelib

2021-02-17T04:28:00Z

Ever since Ansible Collections got introduced, ansible-galaxy collection install had to somehow figure the whole dependency tree that it's supposed to download and install. The code we had rather entangled. But things are going to change starting ansible-core 2.11. And here's how.

One of the items planned for ansible-core v2.11 was improving ansible-galaxy collection CLI 💻. The first thing needed was making possible to upgrade collections when using the install subcommand without requiring --force or --force-with-deps. This is something people have been wanting for quite a while but wasn't possible for roles.

Then, we also wanted to introduce an additional ansible-galaxy collection install [ -U | --upgrade ] option. And we also considered working on the new ansible-galaxy collection remove subcommand but never had time to complete this stretch goal¹.

Another thing on our radar was caching HTTP responses to Galaxy API so that the dependency resolution process could become dramatically faster.

Jordan, Sloane and I formed a feature team to work on this. We decided that we'll try to cut the subtasks one per person to spread the load somehow. Jordan was to work on caching, Sloane was assigned to do the --upgrade task and I was supposed to work on updating the bare install command to make it not require --force (and --force-with-deps for that matter) when there's a need to update the already installed collections.

I was almost unfamiliar with this part of ansible-core so I needed to get myself familiar with it by starting with exloring the pointers of my colleagues on what functions will likely to need updates. What could possibly go wrong? Well, as I was going deeper and deeper down the rabbit hole, I realized that there was a lot of complexity in the existing code and we basically had a rather simplistic dependency resolver that looked like a yarn of leaky abstractions 🤯. It was hard to reason about what strategies it follows to get all the transitive dependencies for collections requested to be installed or downloaded. At the same time, I remembered that there is this other prominent project in the Python ecosystem — pip — that recently got a fresh out of the oven dependency resolver resolvelib It's a third-party library that pip bundles but it's also freely available for use via pip install. My buddy Pradyun has been involved with this effort (the pip one) for about four years so I had somebody I could ask dumb questions about the dependency resolution :)

And so the idea to replace the dependency resolver was born. Instead of patching a few places in the old code here and there, I thought why don't I ~~overengineer this task and refactor the whole thing~~ improve the maintainability of the subpackage dedicated to managing collection CLI subcommands!

I must say that my enthusiasm to ~~break all the things~~ refine a whole bunch of already working code was met with a lot of suspicion within the broader Ansible Core Engineering team, at first. This additionally meant introducing a new runtime dependency — something that we almost never do. We now have a good mechanism to help OS packagers seamlessly bundle runtime dependencies, though².

I faced with a challenge — I knew that the idea was good and now I had to convince others that it's not as crazy as it may seem.

I switched into the research mode, looked into what interfaces and hooks resolvelib requires and came up with a tiny 225 LoC long proof-of-concept. I even wired the demo into GitHub Actions CI/CD so people see the result instantly. After that, when folks saw how easy it is to connect resolvelib and delegate the resolution correctness responsibility to it, the team agreed that this refactoring would be useful and we should proceed.

Meanwhile Jordan was working on his caching task. So while I was busy figuring out where to stick resolvelib into our spaghetti, Jordan submitted the API HTTP request caching PR and it got merged without any problems.

The resolver replacement work was so fundamental that it turned out to block virtually everything else related to our ansible-galaxy CLI UX improvements. This was no longer just my task. Yes, I was making most of the design for the new architecture but I got just enormous amount of help getting this to the finish line. And I enjoyed this collaboration so much!

It wasn't just throwing old code away and adding the new one in place. One of our main objectives was to keep the behavior as close as possible to what the old code did. We've identified a lot of reduntant tests that could be removed, rewrote some of the unit tests into integration tests. We've also identified a ton of gaps in the test coverage which we filled in with many new tests (yaaay! 🙌). Sloane also did a lot of manual behavior verification and testing 👏.

resolvelib and fancy design patterns

I mentioned earlier that resolvelib was easy to integrate and even linked that extremely short PoC. This creates an illusion that it could be a "5-minute patch" but it totally wasn't.

resolvelib requires one to implement an interface they call "provider" with the following hooks:

identify(requirement_or_candidate) — returns a unique identifier for the package (FQCN in our case)
get_preference(resolution, candidates, information) — makes a sort key determining the "importance" of a certain requirement
find_matches(requirements) — returns all candidates matching the given requirements
is_satisfied_by(requirement, candidate) — double-checks the correctness of the candidates resolver chooses
get_dependencies(candidate) — retrieves all the direct requirements that given candidate has

This doesn't look too complicated, does it? That's because resolvelib really doesn't care what your requirements and candidates are for as long as you keep interfacing with it via the same data structures.

This also means that the resolver doesn't know where to get the info about the requirements and the candidates beyond the data you provide to it by implementing these hooks.

So we needed to implement talking to Galaxy API, taking into accont more than one Galaxy-like server as a source for retrieving collections. We needed to take into account non-Galaxy provided artifacts like direct URLs to tarballs or Git repos, or local files and folders.

This all could easily increase the complexity so I introduced the concepts of a concrete artifacts manager, and a facade for talking to multiple Galaxy APIs and other metadata sources (including the artifacts manager). The artifacts manager is responsible for downloading and caching the artifacts (if they are not local) as well as retrieving (and caching) their metadata. It also has an alternative constructor that can clean up the cache directory upon exit. Both objects are initialized once (at the beginning) and are passed to the consumers as a dependency injection.

Most of the packaging ecosystems are rather simple. They have packages with the content of one "atom" inside the artifact. Ansible Collections are mostly like that but there are additional cases which make everything substantionally more complex. One of the primary use-cases that differ is SCM-based collections — they may have one collection in the root of the repository but also in a certain (user-defined) subdirectory. Moreover, SCM targets may have multiple collections inside the same repository (in a namespace subdir that also can be nested as defined by the repo creators). To solve this, we mark Git targets as "virtual collections" during the dependency resolution. The artifacts manager downloads them into a temporary directory and marks that directory a single dependency of such a "virtual Git collection"). If there's subdirs, we do the same "virtual collection" trick with them (except unpacked dirs don't need to be copied into cache, the manager just holds their real paths in memory). These "virtual collections" are very helpful during the resolution and are skipped on the install step (after the resolution is complete).

Fin.

Well, that's about it. 3–4 months into experimentation, development, testing, polishing and reviews, days before the feature freeze, and the feature is in devel!

Based on the refactoring, Sloane was able to complete her work on the --upgrade option and it got merged too.

Feedback, please 🙏

If you are an end-user who uses ansible-galaxy collection [download|install|list|verify] subcommands, please make sure to tell us how well we managed to mix refactoring with the feature development this time. Hopefully, we've squashed all the bugs already 🤞 but we missed anything — let us know! 🖖

The attempt to implement it revealed that we need more design discussion to define how exactly ansible-galaxy collection uninstall is supposed to work.↩
The downstream packagers can now just drop the external dependencies into lib/ansible/_vendor/ transparently instead of packaging them separately, starting ansible-base v2.10.↩

Et Tu Brutè? Use Travis CI for FOSS no more.

2020-11-13T01:51:00Z

Something sadly¹ expected has happened at the beginning of the last week. Travis CI — a pioneer in the field that brought automated testing seamlessly integrated with GitHub to ~~masses~~ many open-source projects on scale — announced that they are migrating all of the public projects that previously got the service for free to a trial plan with a limited amount of toy credits to use².

Ever since they were acquired³ by a company with a rather shady past⁴, Travis CI kept going down this path. They've been shrinking the resources again and again over the past few years and it seems this sort of outcome was inevitable, expecially once they've suddenly layed off a lot of senior engineers⁵.

Of course, they've made a rather pathetic attempt to assure everyone that they'll continue to support open source. But who are they kidding? I bet most of the maintainers have better things to do then go begging support for a bunch of free credits every now and then just to keep things runing. Folks keep underestimating the FOSS maintenance effort and it even seems like projects using Tidelift may be inelligible⁶.

Now What?

Even before the acquisition there's been signs that Travis CI wasn't doing well. There are a lot of articles on setting up other CIs like Azure Pipelines or GitHub Actions CI/CD Workflows. Most of the alternative options provide a comparable experience but may have slightly different ways of being set up. There are also even more powerful CIs like Zuul that are available to significant FOSS projects.

So can we dump Travis CI yet?

Yes, we totally can do that! Should we, though? I'm personally planning to stop advocating for using Travis CI if it's not necessary for a given project. Back in the day, I even contributed a GitHub Pages deployment provider into their dpl project so I feel a little nostalgic... I used to give linting CI set up tasks to my mentees based on Travis. But now, I don't want to advertise a FOSS-unfriendly lock-in so I'll switch to GitHub Actions CI/CD Workflows.

There's one case when I may need to consider using Travis CI additionally to other systems — there are cases when I'd want to run tests in environments (architectures) that none of the other CIs provide. But this will be decided on per-project basis.

Huh. I guess that's all I've been wanting to write. 🖖

A lot of people got upset because of this Travis CI fuckup.↩
Travis gave everyone 10K credits and suggested that people would need to switch to a paid plan after that.↩
Travis announced the acquisition by Idera on Jan 23, 2019.↩
Some people on HN seem to have previous experience with Idera ruining their acquired businesses. Also, they've announced all sorts of commitments like keeping the Travis Foundation alive and now, almost two years in, that domain is dead and googling doesn't even find any mentions of it.↩
Idera removed many essensial employees without a warning and folk on Twitter call this the last nail in the coffin.↩
It's not yet clear but people on Twitter speculate that Tidelift-baked FOSS projects may not get free credits ↩

Hello Website. Again.

2020-11-01T23:06:00Z

Once upon a time, I used to have a blog. I've lost it at some point. But here I am again. Starting over... As if this time I won't have excuses to postpone writing new posts. The definition of insanity, huh?

Anyway. It's now time to start over. The old blog used WordPress and thus required a web-server with some software like nginx and php-fpm. What sounded necessary back in the day seems ridiculous today. Personal blogs don't need to generate pages on flight, or hit a separate database. All that's needed is some static site generator. And so I chose Lektor for this purpose. It's pythonic and very well customizable — just what one needs to run a blog that can be published to GitHub Pages.

Urgh... That's a pretty long backstory so I'll stop here.

Welcome to my blog!