Package description: moreutils

Description: additional unix utilities
This is a growing collection of the unix tools that nobody thought to write thirty years ago.

So far, it includes the following utilities:

• sponge: soak up standard input and write to a file – I use this utility in bash scripts all the time.  When I am processing lots of text data I need to do things like grep for strings in a file and then replace the original file with just the data pulled from the grep.  If you were to try grep string file > file you would end up with an empty file. To get around that you could do grep string file > file.tmp ; mv file.tmp file if you wanted but spong makes it easier With sponge you can do grep string file | sponge file and the resulting file will contain the exact results of your grep.  It works by storing all the grep results in memory until the grep is finished and only then writing them down to the file.  This can simplify many tasks.
• ifdata: get network interface info without parsing ifconfig output
• vidir: edit a directory in your text editor – Say you have a directory full of files you want to rename in ~/Photos/ItalyTrip.   You can use vidir ~/Photos/ItalyTrip and you will get a directory listing of that folder in your default text editor.  You can then use find and replace or any other tool in the text editor to rename the files and folders.   When you exit the editor and save the changes the files and folders in ~/Photos/ItalyTrip will be renamed to match what you did in the text editor.
• vipe: insert a text editor into a pipe
• ts: timestamp standard input – This handy little tool will add a timestamp to data coming from standard input.  I have a series of backup scripts that run nightly and I end each script with echo “$backjob Complete” |  ts > ~/Logs/backups.log if the job finishes successfully.  By piping the echo statement through ts I get the date and times as a preface to the “Complete” statement in the log.  There are other ways to do this but this way ensures that I get the exact time and date that the job finished as opposed to when the job started.
• combine: combine the lines in two files using boolean operations
• pee: tee standard input to pipes
• zrun: automatically uncompress arguments to command
• mispipe: pipe two commands, returning the exit status of the first
• isutf8: check if a file or standard input is utf-8
• lckdo: execute a program with a lock held

Homepage: http://kitenet.net/~joey/code/moreutils/

To install moreutils in any current Debian install or Ubuntu 8.04LTS or newer use:

apt-get install moreutils

Right now, with so many people running the new Ubuntu 9.04 Beta, Ubuntu users are finding themselves in a position of needing to get updates at least daily.   By default, your new Ubuntu install will be set to use an official Ubuntu mirror along with thousands of other users.

One of the nicer things about Ubuntu is that many organizations and Universities maintain Ubuntu mirrors.  This guide will walk you through testing for the fastest mirror/server and selecting that mirror as your source for updates to your Ubuntu installation.

Open the Software Sources applet by clicking System and then Administration and then Software Sources.

Reload

Software Sources

Click to the right of Download from: and select Other.

Select Best Server

Click on Select Best Server in the upper right of the window.

Checking for fastest mirror

It may take a few minutes to test all the mirrors to detemine wich is fastest for your location and ISP.

Best Server Highlighted

When the testing finishes, the fastest server will be highlighted.  Click the Choose Server button in the lower right.

Close Applet

You will now see the fastest server to the right of the Download From label.   Click Close in the lower right.

Checking For Updates

Since you have changed the Mirror, Ubuntu needs to check for updates.  Click Reload and it will check the new repository.

This process can be repeated anytime you notice that Updates are downloading slowly.  Just because one mirror is fastest today, it doesn’t mean that it will be the fastest forever.

Related articles by Zemanta

• Jackalope sighting: first look at Ubuntu 9.04 beta (arstechnica.com)
• Download Ubuntu Pocket Guide and Reference (blogs.vinuthomas.com)
• Clean up unwanted files in Ubuntu – BleachBit (blogs.vinuthomas.com)

I’ve installed the plugins in Ubuntu 8.04 and Firefox 3 and while I get a picture its more like a slideshow than a streaming video.  I don’t know if thats by design, or if its a problem.  The new player can only stream the 500Kbps stream so it may be a bandwidth thing right now.

I’d love to see how much of the wired (non-satellite) Internet capacity in Washington DC is in use today between 11:00am and 2:00pm.

Even if it does not play as well as one could hope, the folks behind this get an A for effort.

Obama Inauguration on Linux and PowerPC Macs! Plus codec details | Ben Waggoner

We kicked the project off this afternoon before the event, so pardon us if it’s a little rough. The only significant limitation is that the SL1 player can only do the 500 Kbps stream – lacking managed code, making a manifest XML parser in JavaScript didn’t seem like something we should start testing only nine hours before the event goes live.

But anyway, thanks to Joseph, Larry, Geoff, Rusty and especially Aaron at Novell for pulling this together so quickly and so well, and Mio at iStreamPlanet for making the last minute encoder tweaks to make it Silverlight 1.0 compatible. On the Microsoft side, Brian Goldfarb and Eric Schmidt did a great job making the call and making it happen. Just go to www.pic2009.org in the morning and the updated player should be available around 6 am EST (11 am GMT).

That something like this could come together on such short notice is a testament to the chops and passion of the Moonlight team and the great platform we’re both implementing. I’m glad we got this chance to demonstrate how serious we are about this collaboration (although I wouldn’t have minded starting a day earlier…).

Update: The Silverlight 1.0 compatible player is low live at:  http://www.pic2009.org/page/content/linuxplayer

Note that this player is only for Silverlight 1.0 and doesn’t work with Internet Explorer. The Silverlight 2.0 player for Windows or Intel Mac is at: http://www.pic2009.org/page/content/live

Seagate has announces, in its Knowlege Base, that the problem is a result of bad firmware and that drive owners can protect themselves by upgrading the firmware for the affected drives. They have also confirmed that the data, while inaccessible to the drive owner, is still intact.   Seagate has released a statement to the folks at TechReport.com saying they will recover the data for customers who experience the failure.

Customers can expedite assistance by sending an email to Seagate (discsupport@seagate.com). Please include the following disk drive information: model number, serial number and current firmware revision. We will respond, promptly, to your email request with appropriate instructions. There is no data loss associated with this issue, and the data still resides on the drive. But if you are unable to access your data due to this issue, Seagate will provide free data recovery services. Seagate will work with you to expedite a remedy to minimize any disruption to you or your business.

There is a bit of a problem with the Seagate KB article though. It assumes that you are a Windows user. They provide a tool to let Windows users scan their drives and discover the model number, serial number, and firmware revision.  However, if you are a Linux user their Knowledge Base article is not going to help you.  Fortunately, this information is very easy retrieve in most Linux installations.  If you are using a Debian or Ubuntu install this method will definitely work.

You can skip down to the end for a simple bash command to get this info, but first I will go through it step by step.

To check the drives, you will need to determine your hard drive device names.  A quick and dirty way to get this information is to check your /dev/disk/by-path folder

~$ ls -l /dev/disk/by-path

You will see all your drives.

On my Desktop I get a list like this:

pci-0000:00:02.1-usb-0:2:1.0-scsi-0:0:0:0 -> ../../sdf
pci-0000:00:02.1-usb-0:2:1.0-scsi-0:0:0:0-part1 -> ../../sdf1
pci-0000:00:02.1-usb-0:2:1.0-scsi-0:0:0:1 -> ../../sdg
pci-0000:00:02.1-usb-0:2:1.0-scsi-0:0:0:1-part1 -> ../../sdg1
pci-0000:00:02.1-usb-0:5:1.0-scsi-0:0:0:0 -> ../../sdm
pci-0000:00:02.1-usb-0:5:1.0-scsi-0:0:0:0-part1 -> ../../sdm1
pci-0000:00:02.1-usb-0:6:1.0-scsi-0:0:0:0 -> ../../sdn
pci-0000:00:02.1-usb-0:6:1.0-scsi-0:0:0:0-part1 -> ../../sdn1
pci-0000:00:02.1-usb-0:8:1.0-scsi-0:0:0:0 -> ../../sdh
pci-0000:00:02.1-usb-0:8:1.0-scsi-0:0:0:1 -> ../../sdi
pci-0000:00:02.1-usb-0:8:1.0-scsi-0:0:0:2 -> ../../sdj
pci-0000:00:02.1-usb-0:8:1.0-scsi-0:0:0:3 -> ../../sdk
pci-0000:00:06.0-scsi-0:0:0:0 -> ../../sdd
pci-0000:00:06.0-scsi-0:0:0:0-part1 -> ../../sdd1
pci-0000:00:06.0-scsi-0:0:1:0 -> ../../sde
pci-0000:00:06.0-scsi-0:0:1:0-part1 -> ../../sde1
pci-0000:00:08.0-scsi-0:0:0:0 -> ../../sda
pci-0000:00:08.0-scsi-0:0:0:0-part1 -> ../../sda1
pci-0000:00:08.0-scsi-0:0:0:0-part2 -> ../../sda2
pci-0000:00:08.0-scsi-1:0:0:0 -> ../../scd0
pci-0000:00:08.1-scsi-0:0:0:0 -> ../../sdb
pci-0000:00:08.1-scsi-0:0:0:0-part1 -> ../../sdb1
pci-0000:00:08.1-scsi-0:0:0:0-part5 -> ../../sdb5
pci-0000:00:08.1-scsi-0:0:0:0-part6 -> ../../sdb6
pci-0000:00:08.1-scsi-0:0:0:0-part7 -> ../../sdb7
pci-0000:00:08.1-scsi-0:0:0:0-part8 -> ../../sdb8
pci-0000:00:08.1-scsi-1:0:0:0 -> ../../sdc
pci-0000:00:08.1-scsi-1:0:0:0-part1 -> ../../sdc1

This technique will not work on USB drives and for our purposes we only care about drives, not partitions so since /dev/sdb1 and /dev/sdb5 etc… are on the same drive we need to whittle down the list a little more.  Your drives could show has /hd* instead of /sd –  just use what you have for the rest of the post. Try this command in your terminal.

~$ ls -l /dev/disk/by-path/ | grep -v usb | grep -v [0-9]$

On my Desktop, this brings our output down to:

pci-0000:00:06.0-scsi-0:0:0:0 -> ../../sdd
pci-0000:00:06.0-scsi-0:0:1:0 -> ../../sde
pci-0000:00:08.0-scsi-0:0:0:0 -> ../../sda
pci-0000:00:08.1-scsi-0:0:0:0 -> ../../sdb
pci-0000:00:08.1-scsi-1:0:0:0 -> ../../sdc


Those are my two internal hard drives so those are the ones I need to query to find out their their information to compare to the Seagate site.
Now at this point we want to remember our drive devices.  Just make a list of the last three letters prefaced with /dev  so for me.

We can do that in the shell by using sed and lets sort them while we are at it.

~$ ls -l /dev/disk/by-path/ | grep -v usb | grep -v [0-9]$ | sed "s/^.*\./\/dev/g" | sort

This gives us:

/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde

The easiest way I know to get the information we need is by using the hdparm tool that is built into most Linux systems.  Two different hdparm commands will provide the information we want.  If you want to see a lot of information in a very pretty format try (substi:

~$ sudo hdparm -I /dev/sda | more

This gives us all kinds of data about that drive:

/dev/sda:

ATA device, with non-removable media
Model Number:       ST3750640AS
Serial Number:      5QD2DLC0
Firmware Revision:  3.AAE
Standards:
Supported: 7 6 5 4
Likely used: 7
Configuration:
Logical        max    current
cylinders    16383    16383
heads        16    16
sectors/track    63    63
--
CHS current addressable sectors:   16514064
LBA    user addressable sectors:  268435455
LBA48  user addressable sectors: 1465149168
device size with M = 1024*1024:      715404 MBytes
device size with M = 1000*1000:      750156 MBytes (750 GB)
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16    Current = 16
Recommended acoustic management value: 254, current value: 0
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
Enabled    Supported:
*    SMART feature set
Security Mode feature set
*    Power Management feature set
*    Write cache
*    Look-ahead
*    Host Protected Area feature set
*    WRITE_BUFFER command
*    READ_BUFFER command
*    DOWNLOAD_MICROCODE
SET_MAX security extension
*    48-bit Address feature set
*    Device Configuration Overlay feature set
*    Mandatory FLUSH_CACHE
*    FLUSH_CACHE_EXT
*    SMART error logging
*    SMART self-test
*    General Purpose Logging feature set
*    SATA-I signaling speed (1.5Gb/s)
*    Native Command Queueing (NCQ)
*    Phy event counters
Device-initiated interface power management
*    Software settings preservation
Security:
Master password revision code = 65534
supported
not    enabled
not    locked
frozen
not    expired: security count
not    supported: enhanced erase
Checksum: correct

The information we really want is in the top section:

Model Number:       ST3750640AS
Serial Number:      5QD2DLC0
Firmware Revision:  3.AAE

If you only have a couple of drives you can just repeat the hdparm -I /dev/DRIVE for each drive and then use that information to compare to the affected model numbers listed below.  With as many drives as I have, I want to use a different hdparm parameter.  This one outputs much of the same information, but in a more condensed format:

~$ sudo hdparm -i /dev/sda

This time we get back:

/dev/sda:
Model=ST3750640AS, FwRev=3.AAE, SerialNo=5QD2DLC0
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=?16?
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1465149168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7

As you can see, the line starting with Model= has all the information we need.  The rest of the output is nice, but not needed.   We can clean this up and get just what we need by building this command:

TEMPLATE:  sudo hdparm -i /dev/[hs]d[DRIVES] | grep -A 2 dev | sort | uniq | grep Model

In the above line you should replace [DRIVES] with your drive designation letters.  In my case, I want to see the results of /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd, and /dev/sde. So I will replace [DRIVE] with [abcde] in this command:

~$ sudo hdparm -i /dev/[hs]d[abcde] | grep -A 2 dev | sort | uniq | grep Model

This gives me:

Model=ST3750640A, FwRev=3.AAE, SerialNo=5QD3VYA7
Model=ST3750640A, FwRev=3.AAE, SerialNo=5QD4E17G
Model=ST3750640AS, FwRev=3.AAE, SerialNo=5QD1MCM8
Model=ST3750640AS, FwRev=3.AAE, SerialNo=5QD2DLC0
Model=WDC WD7500AACS-00ZJB0, FwRev=01.01B01, SerialNo=WD-WCASJ1008654

Using this information, we need to check the list of Seagate models.  The  models listed below are current as of January 18, 2009.  You may want to visit the Seagate page here in case they add addtional affected models.

If you are the proud owner of one of these models you will want to follow these instructions from Seagate:

If you have one of the models listed above, Customers can expedite assistance by contacting Seagate via email. Please include the following disk drive information: model number, serial number and current firmware revision.  We will respond, promptly, to your email request with appropriate instructions.

Or you can call Seagate Support at 1-800-SEAGATE.  Please be prepared to give the serial number of your drive as the solution depends on knowing the exact serial number.

For a list of international telephone numbers to Seagate Support and alternative methods of contact, please access http://www.seagate.com/www/en-us/about/contact_us/

A couple days ago this came out over at TheInquirer.net:

Windows 7 is enough to kill Linux on the desktop

FOR THE PAST three years I have been a Linux fan-boy using Ubuntu most of the time and Windows XP when I needed to play games or run CS desktop lay out stuff.

In a bid to focus my bile on something other than Apple for a bit I decided to play with the new Windows 7 beta. I was disappointed. It was pretty good and, if I am right, could result in the move away from Voleware to Linux and OSX being stopped in in its tracks….

And then that was addressed on a ZDNet.com blog:

7 reasons why Windows 7 will not wreck Ubuntu

One of our competitors has a review of the Windows 7 beta which claims that desktop Linux is doomed — doomed!

Not exactly. Not even approximately.

First let’s understand what 7’s target is. It’s not so much desktop Linux as a particular Linux distro — Ubuntu — that targets the desktop…

The smart money is on those who understand that this really only applies to the class of PC users who actually give a damn about which OS is running on their system.  Sadly, at present, thats a tiny fraction of users.  People use what they are given or what they used in school or at the office.  Those of us who shuttle back and forth between multiple OS’s or even multiple incarnations of the same OS will continue to do what we always have and select the best tool for the job.

If I want to use Dreamweaver, I load a Windows XP virtual machine on my Ubuntu Desktop and use it since, in my opinion, XP is the best platform for using Dreamweaver for me – personally.  If I could run OSX in a VM and use Dreamweaver there, my opinion could change.

If I was constrained through my employer to only use Windows as my primary desktop, I would stll have Ubuntu running in a virtual machine for those times when I need a robust shell like bash to do what needs to be done.

With current PC’s shipping with lots of RAM and big hard drives there is no longer any good reason to limit yourself to a single operating system.

One of the big things that I think will help modern Linux users is the default Ext4 behavior for allocation of space on the file system.

Most file systems create files by filling the requested space with zeros.  As anyone who has used BitTorrent to download an ISO knows, this can be time consuming as the system creates a 700mb file full of zero’s that will only be overwritten as the real file is downloaded into that space.

Ext4 handles preallocation more efficiently.  Using BitTorrent to download that ISO onto a volume formatted with Ext4 results in the file system simply “Reserving” the of 700mb contiguous (if possible) space but doesn’t actually write anything to the drive until you start using the file.

As more people use iTunes to download TV shows and movies or as people do more video editing the size of the files they work with get bigger and bigger.  Ext4 addresses this problem and keeps Linux ahead of the curve for users who need to work with lots and lots of data.

OK, OK… so how will the end user benefit from this EXT4 filesystem? Well, first of all, the whole system will be much faster and more reliable compared to one with EXT3 (I guess that some of you still remember the Firefox/Ext3 issue), it will boot faster (the current article proves that) and it’s able to handle files with sizes of up to 16 TB (terabytes). But these are just a few of the features brought by the fourth extended filesystem, for more details you can check the Wikipedia entry for EXT4.

via Ubuntu 9.04 Boots in 21.4 Seconds – With EXT4 as the default filesystem. – Softpedia.

Sadly, this April’s  9.04 release of Ubuntu will only support Ext4 installs via the Alternative/Text based installer.  It won’t be until October’s 9.10 release that Ext4 will become the default file system format for the main Ubuntu installer.

I don’t feel like messing with my CSS to make it render properly here so I did a screen shot.

Click here to see the HTML itself.

 Its a silly thing and I cannot think of a single situation in which I would have a legitimate need to convert a PNG to HTML but it’s nice knowing I can if I want. :)

Png2Html written by Geoff Holden ( http://www.engr.mun.ca/~holden/png2html.html )
Permission is  granted to copy, distribute and/or modify this document  under the terms of the GNU Free Documentation.

If you want to play with this in Ubuntu or other Debian based distros just install it with

sudo apt-get install png2html

 You can get instructions on using it via man png2html after the install.

Google Browser Sync for Firefox is an extension that continuously synchronizes your browser settings – including bookmarks, history, persistent cookies, and saved passwords – across your computers. It also allows you to restore open tabs and windows across different machines and browser sessions.

If you have limited your search for add-ons for Firefox to the official Firefox sit at https://addons.mozilla.org you are missing out on some cool stuff. I make a point of checking out the Google Labs site every month or so to see what those smart folks at Google are up to. I tend to avoid beta or experimental software unless it is designed to address a real need that I have or unless its just really really cool.

Over the last few months I have added a third and then a fourth computer to my list of often used PCs. I have my main PC at home, my main PC at work, a work owned laptop, and I use one of the laptops we have around the house. I do a lot of work on the web and mail servers at the office at all hours of the day or night and I need to be able to do those things quickly and easily no matter which machine I am using. My biggest problem involved keeping my bookmarks synchronized along at LEAST my home and work PC’s. Keeping the laptops synced was less important but still a hassle.

I tried a few different add-ons that I found through the Mozilla site. They all had problems. I had given up and was using a bash script to just copy my bookmarks.html file up to a website nightly. I would then download it to another machine, blow away all my bookmarks, and then import the latest version. It was not a horrible hardship since with practice, this can be done in under two minutes. The biggest problem I had was remembering which machine I had added the new link to and often I would need a link now and stopping to do a new import would destroy my chain of thought.

Google Labs has come to the rescue. If you have a Google account and more than one PC, I highly recommend the Google Browser Sync to keep your bookmarks, passwords, and cookies synchronized between both machines. Google has put together a fairly extensive FAQ for the Browser Sync component.

This tool will let you use Google as a encrypted storage location for your Firefox bookmarks, history, cookies, and stored passwords. It’s such a wonderful thing to browse to a website at home that I logged into at work and find myself still logged in. As Google points out, even if you only have one PC, this tool is an excellent way to make sure that your bookmarks are always backed up. Like most Firefox Add-ons, this tool works with Firefox 1.5 and newer regardless of your platform so you can use this to synchronize between a Windows and Linux system or Linux and Apple, or Windows and Apple.

While the extension runs beautifully once its up and running, I have a few tips that will help anyone who plans to use this on more than one PC. My first attempt resulted in a huge mess in terms of my bookmark organization. I had named some bookmark folders and bookmarks with slightly different names on various PCs. The Google Browser Sync tool synced everything so I I had a lot of duplicates. I uninstalled it, and did a little reorganizing, and started over and it’s been peachy ever sense.

But, on the bright side, Firefox users still outnumber IE users by 3 to 1.