Zero Rating

AT&T wants companies to pay to “sponsor” their zero-rated data. This is the obvious violation of net neutrality. Even if the zero-rating looks good for consumers in the short run (because they can watch video without counting it towards their usage), it is bad for consumers in the long run. Any new video service will count against consumers’ data caps, giving that company a disadvantage. They can only compete if AT&T, at their sole option, chooses to allow them to sponsor zero-rating, and if AT&T charges them the same rate. And even then, this idea of pay-for-access is terrible for many other reasons.

Raspberry Pi 3 Stratum 1 NTP Server (Ubuntu)


I’m in the U.S., so I ordered only the GPS board and case from Uputronics to save on shipping.

Other GPS HATs (and cases) can be used. Consult the Stratum-1-Microserver HOWTO for GPIO pin changes in step 12.

1. From Ubuntu Pi Flavour Maker, download the Ubuntu Server Minimal 16.04 image. Note that this is only available using BitTorrent (for bandwidth reasons).

2. Write the image to a Micro SD card.

3. Insert the SD card into the Raspberry Pi. Stick the heatsink to the processor. Assemble the case with the Raspberry Pi in it as you go. Connect the GPS antenna and place it near a window. Connect a keyboard, mouse, and monitor. Power up the Raspberry Pi.

4. Login with “ubuntu” as the username and “ubuntu” as the password. Set your own password:

5. Become root:
sudo -s

6. Generate the missing locale:
locale-gen en_US.UTF-8

7. Disable a broken service:
systemctl disable ureadahead

8. Resize the partition to fill your Micro SD card:
fdisk /dev/mmcblk0
“Delete the second partition (d, 2), then recreate it using the defaults (n, p, 2, enter, enter), then write and exit (w).” — Ubuntu Pi Flavour Maker FAQ

9. Set your time zone:
dpkg-reconfigure tzdata

10. Set your hostname:
vi /etc/hosts
vi /etc/hostname

11. Actually, I think we want to leave this alone. I can’t reproduce any problem, and keeping fake-hwclock, as designed, makes the clock far less wrong on initial boot (before ntpd steps the clock). Disable fake-hwclock, which otherwise breaks PPS in NTP at boot:
systemctl disable fake-hwclock

12. Disable Bluetooth, as we need the UART for GPS:
echo dtoverlay=pi3-disable-bt >> /boot/config.txt
echo enable_uart=1 >> /boot/config.txt
echo dtoverlay=pps-gpio,gpiopin=18 >> /boot/config.txt
systemctl disable hciuart
apt -y purge bluez bluez-firmware

13. Set maximum performance for consistent timing:
sed -i "s|$| nohz=off|" /boot/cmdline.txt
systemctl disable ondemand
echo 'GOVERNOR="performance"' > /etc/default/cpufrequtils

14. Install software:
apt update
apt -y dist-upgrade
apt -y install cpufrequtils gpsd gpsd-clients ntp pps-tools

15. Configure gpsd:

sed -i 's|DEVICES="|\0/dev/ttyAMA0 /dev/pps0|' \
sed -i 's|GPSD_OPTIONS="|\0-n|' /etc/default/gpsd
mkdir -p /etc/systemd/system/ntp.service.d
cat >/etc/systemd/system/ntp.service.d/gpsd.conf <<EOF
mkdir -p /etc/systemd/system/gpsd.service.d
cat >/etc/systemd/system/gpsd.service.d/stationary.conf <<EOF
ExecStartPre=/usr/bin/gpsctl -t 'u-blox' -b -x '\\\\x06\\\\x24\\\\xFF\\\\xFF\\\\x02\\\\x03\\\\x00\\\\x00\\\\x00\\\\x00\\\\x10\\\\x27\\\\x00\\\\x00\\\\x05\\\\x00\\\\xFA\\\\x00\\\\xFA\\\\x00\\\\x64\\\\x00\\\\x2C\\\\x01\\\\x00\\\\x3C\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00' \$DEVICES

16. Configure ntp
cat >/etc/ntp.conf <<EOF
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift
leapfile /usr/share/zoneinfo/leap-seconds.list

statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# Specify one or more NTP servers.

# Stratum 0 (GPS)
server minpoll 1 maxpoll 1 prefer
fudge refid PPS

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See for
# more information.
pool iburst preempt
pool iburst preempt
pool iburst preempt
pool iburst preempt

# Use Ubuntu's ntp server as a fallback.
pool preempt

server noselect
fudge time1 0 refid GPS

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html
# for details. This page might also be helpful:
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict ::1

# Needed for adding pool entries
restrict source notrap nomodify noquery

17. Reboot:

18. Resize the filesystem:
sudo resize2fs /dev/mmcblk0p2

19. Check that things look right:
ntpq -p

20. Wait ~24 hours for everything to stabilize. You want to allow for the drift to be calculated. Once the PPS offset is tiny (e.g. 0.010 or so), you know things are good.

21. At this point, wait for midnight to roll around so the stats are rotated. Then wait at least another 4 hours.

22. Calculate the correct offset for the GPS serial data:

awk '/(SHM\(0\)|127\.127\.28\.0/ { sum -= $5 ; cnt++; } END { print sum / cnt; }' /var/log/ntpstats/peerstats

23. Add that value from the GPS’s time1 in /etc/ntp.conf. If this is the first time you’re doing it, the existing value is zero, so just use it directly.

24. Restart ntp:
sudo systemctl ntp restart

25. The offset on the GPS line should now be less than 3 or so. You can repeat steps 21 through 24 if want to try to get closer.

26. Remove noselect from the GPS line in /etc/ntp.conf:
sudo vi /etc/ntp.conf

27. Restart ntp:
sudo systemctl ntp restart


ISP Traffic Prioritization

This was originally posted as a Slashdot comment. It discusses the idea of prioritizing traffic in an ISP environment, ideally using markings generated by the customers.

I do network engineering at an ISP. We are small, though I have discussed these things with my peers at larger networks.

Once you scale above a very small network (like your home connection), allowing congestion isn’t really okay in practice, even with QoS. When I say it’s not “okay” here, I’m speaking purely technically.

It might be possible to let networks congest somewhat if you had a large amount of elastic traffic that you could reliably identify. Netflix, for example, could meet these criteria. But that’s not okay politically; that’s an example of why net neutrality is good!

QoS in carrier networks is only useful for priority (de-)queuing of traffic to reduce latency and jitter. For example, real-time voice or video traffic could benefit. This is where it’d be nice to actually be able to honor user traffic markings.

It’s not (currently at least) practical to make the decisions on a flow-by-flow basis in the core of the network (which is what your proposal would require). This is a hardware scaling issue. To be clear, tracking flows statistically is okay at scale. ISPs do plenty with NetFlow/sFlow. But taking an incoming packet, assigning it to a flow, and marking it appropriately, for every packet, in real time is the scaling challenge.

The following approach would scale perfectly in trusted CPE (ONT/cable modem) or reasonably well in a DSLAM (for DSL). Give each user (for example) two queues. Honor the incoming DSCP markings. Put a small, but reasonable, limit on the size of the priority queue; overflowing traffic gets remarked and placed into the non-priority queue. Then, honor markings through the rest of the network.

There are a few problems with even this approach. First off, there are going to be users who legitimately create more high priority traffic than any limit that’s acceptable across the board. Is it okay to charge them for a higher limit? If not, how do you avoid gaming the system? If yes, won’t that incentivize ISPs to set the limit to zero and charging for all priority? Is that okay? If so, what fraction of people will request and pay for priority in that world? Will that be enough to encourage application developers to mark traffic appropriately? Or does this just degrade into our current zero-priority Internet?

Second, this only gets you one direction (upload). To handle the download direction, you’d need to honor priority bits on your upstream and peering links. But there, you can’t trust the markings (unless it’s a 1:1 peering link and you are guaranteed your peer implements a compatible policy at their incoming edge), at least without policing. Policing the queues there is easy, but gives you terrible results in real life. If the limit is exceeded with traffic that “should not have been” marked priority, it will destroy the prioritization of “legitimate” priority flows by forcing some fraction of their packets into the non-priority queue. If you accept all (or just a high enough fraction of) incoming traffic as priority traffic, then you have destroyed the prioritization yourself. If you try to mark flows per IP/customer, we’re back to that scaling problem.

It might be possible to do something that involves tracking flows at the customer edge and using the incoming markings for the downstream direction. But this is only prioritizing in the last mile. At best, this is a lot of work for very little benefit.

Gun Control Ideas

This is an excerpt from a Facebook discussion about gun control.

First, I recommend everyone read this link, which started the discussion:

[In the comment thread, someone suggested licensing, registration, and insurance. Another person then made a comparison to cars.]

I think one can make a reasonable argument that training and licensing is keeping with the “well-regulated” spirit of the amendment. That said, it will do nothing to stop people who steal the guns from their relatives (like multiple recent mass shooters), or this last guy who was licensed and trained.

I don’t think the insurance thing would help, unless you’re also proposing a radically different type of insurance than we’re used to. If someone steals my car, I’m not liable for what they do, and thus my liability insurance is irrelevant. If I use my car to kill a bunch of people and myself, that’s criminal and my insurance doesn’t have to pay; I’m liable, but that’s irrelevant if I’m dead and have no assets.

Here are some ideas I like (numbered for identification, not for priority or order):

1) Spend as much money as it takes to investigate each-and-every NICS denial and prosecute 100% of those which were legitimate denials. Every. Single. One. Right now, we prosecute less than 0.1%! Can you imagine any other scenario where the government literally directly says, “No, what you(r customer) just tried to do is illegal.” and then prosecutes essentially nobody?

2) Fix issues where certain data (especially mental health data) that is already supposed to make it into NICS is not making it into NICS in some cases. (I don’t recall the details of this problem off the top of my head.) But, we need to be careful and ensure that simply seeing a counselor doesn’t result in a gun rights revocation, or it will disincentivize people from getting help.

I don’t know how much the above will help, but it’s unreasonable to ask for more laws when the existing ones aren’t being enforced.

Other measures might be okay, but they definitely need safeguards. For example:

3) Require NICS checks for non-family private transfers. But, only real sales, not loaning your gun to your friend at the range. And, if you call NICS and they can’t process with X minutes (say 5 or 10) or you get no answer having called twice waiting Y minutes (say 3 or 5) in between, that counts as an approval. This rule is critical, because otherwise the government can enact a de facto gun ban by simply defunding the background checks.

On the other hand, laws need to respect civil rights and be evidence-based:

4) Eliminate Gun Free Zones. This is a violation of people’s rights. Research shows it makes things worse, not better. Criminals who are going to commit murder don’t care about another charge, especially if they’re going to kill themselves anyway. These laws just disarm potential victims.


Suggestions to ban only rifles (or subsets thereof) are a bad idea. Not only do more murders use knives than rifles, but more murders use no weapon at all (fists, etc.) than rifles: Rifles are a very small portion of the murder problem. And, if we outlawed rifles but not handguns, then I suspect a lot of the rifle murders would just become handgun murders.

The typical bans suggested are: 1) all guns, 2) handguns, 3) “scary-looking” rifles (assault weapons ban). It’s well-established (even by gun control supporters) that the assault weapons ban accomplished nothing.


I think the most important question that people ask themselves is: what problem do I want to solve?

If the question is, as was brought up, “The legal system should honor the original meaning of the 2nd Amendment.”, well, I think SCOTUS needs to go a lot further (further than even I would support) in what is allowed. To justify the current limits on paper, I think we’d have to amend the constitution to limit the 2nd amendment.

If the question is, “Guns are inherently a dangerous tool and should be regulated accordingly.”, I personally want enforcement (since we have essentially none), plus I’m at least open to (and think I want) a few more limits. When you get outside of sound bites, I found that many other pro-gun rights people I talk to agree with me. The other day, I saw some poll numbers that suggests a majority of the public supports the additional limits I mentioned. Granted, I also want to repeal one limit, too.

If the question is, “What can we do about mass murders?”, I honestly don’t know. Not making the killers infamous might help a little. Tackling social problems is probably the best approach. Any popular proposed gun rule I’ve heard short of a total gun ban (to the extent it would even be possible) would not have stopped one or more recent mass shooters. And, there’s a risk some might move to bombs; it wouldn’t take many people moving to bombs to end up with the same total death toll, since bombs are even more deadly. Rights aside, I cannot support a complete ban, because I believe it will increase innocent deaths. We would trade less mass murder victims for more routine crime victims.

If the question is, “What can we do about total crime deaths?”, I think the answer is similar, but also involves approaching drug use in a public-health way rather than a criminal-law way. Tobacco is a public health problem, but Philip Morris isn’t having shootouts with its competitors.

If the question is, “What can we do about suicides?”, I’m not sure. I really, really want to reduce depressive suicides. But if people were driving their cars off cliffs, I wouldn’t want to ban cars. I’m not sure if there’s a middle ground with guns that helps. But I do think we need to spend more money, time, and attention on mental health. Our local hospital is booked up a month or two out; that’s completely unacceptable for people who need help now.

If the question is, “How can I get rid of all guns, because I don’t think anyone should have guns.”, my answer is that you don’t have the right to decide that for me.

Nexenta Upgrade Failure

I’m documenting this mainly for myself, but if you’re ending up here based on a Google query, I hope it helps!

I tried to upgrade our Nexenta storage system (currently running 4.0.3FP3). After apt-get downloaded packages, I received this error:
Download complete and in download only mode
Upgrade is in progress. Please DO NOT interrupt...
Creating Upgrade Checkpoint...
Feb 03 19:13:23 EXCEPTION: FormatError: Failed to parse menu.lst: section content not complete
Uncaught exception from user code:
com.nexenta.nmu.FormatError: Failed to parse menu.lst: section content not complete
at /usr/perl5/5.12/lib/NZA/ line 731
NZA::NMUUtil::_mark_rootfs('syspool/rootfs-nmu-008', 0, '') called at /usr/perl5/5.12/lib/NZA/ line 817
NZA::NMUUtil::clone_rootfs() called at /usr/bin/nmu line 526

Nexenta tech support found that the issue was empty BOOTADM blocks in /syspool/boot/grub/menu.lst:
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
#---------------------END BOOTADM--------------------

The fix is to remove those and run bootadm update-archive -v.

If I understood correctly, the cause may have been using beadm destroy in the shell instead of setup appliance checkpoint ... in nmc.