Thursday, February 7, 2013

Example: Lazy Map and Least Recently Used (LRU) Map

In this post, I will demonstrate both an Apache Commons Collections Lazy Map and an Apache Commons Collections Least Recently Used (LRU) Map used in concert.

I had many ( and I mean many ) examples of a Lazy Map. The first example used Google SOAP Services for data aggregation, I wrote it almost three years ago. Sadly, the life cycle of the Google's SOAP services and my timeliness didn't mesh. By the time I was ready to post it, it no longer existed. Then I decided to use Google Base, so I spent a month writing it and decided the article was overly complicated and was more about Google Base then the Lazy Map. Frustrated, I wrote something that used HSQLDB, but when I was done it was over the top ( I'm still laughing about the complexity ). I put the whole project on the back burner for long time, and then I found inspiration again with a simple requirement while working at IBM. This was the first time I was able to use a LRU ( Least Recently Used ) map in a real world environment ( not just an example ). Enjoying the act of learning, I became rather excited to use it. This was a great opportunity to see the LRU map in action and explore the Lazy Map.

You may have heard the proverb if all you have is a hammer, everything looks like a nail. It means, single-minded people who have limited knowledge usually apply only one solution. Understanding the fundamentals of both the Java Collections Framework and the Apache Commons Framework will boh widen your skills and make you a far better asset than someone who does not. I knew of the LRU map and the Lazy Map, but I never used either in a real world scenario. I just didn't have a need for either of them. I’m sure someone without knowing about it would be able to write something in a day or worse two. Conversely, all I had to do was read the documents and I was up and using it in about an hour. If you are like me and enjoy learning new things, you can image my excitement when I dug deep into this class.

Naturally I can’t go into the details of the requirement that drove me to this class, but the objective was to keep track of values statically with keys. Arguably It could have been done with a static map or some kind of Singleton or any combination of the two. However, I had to get it done fast, the values (results of the operation) were expensive to manufacture, large, and the activity on the object would focus on a few keys at a time. Being concerned about space, I calculated the memory usage per value and set a limit to the size of the map (part of the LRU Map behavior). One of the well-designed aspects of the entire Apache Commons Framework is its use of the decorator pattern and the factory pattern. Unfortunately, the factory interface is completely un aware of the key. Making it useless for my needs. With the decorator pattern, which has access to the key, I could wrap the LRU Map with a Lazy Map. Expanding the functional characteristics a simple HashMap. The Lazy Map can "fill in" the expired values automatically and the LRU can "expire out" infrequently accessed values. Using a static Lazy Map, on the other hand, could, if unbounded, easily consume large amounts of memory.

I built an example that uses these two objects and will fetch US Stock Exchange Market Data (specifically NASDAQ) from Yahoo. Like all Maps, you need to be aware of Thread Safety  and use them with discretion. There are plenty of examples, explanations of concurrency and scalability issues with Maps, if you Google them. If you examine my example closely, you might take note of a serious short coming. I don't want to give it away, it might be fun for some to find it. Ill explain it near the end, and give some examples on how to over come it. I hope you enjoy this walk through let me know if you find it helpful.

The developers of Apache Commons Collections framework use the decorator pattern extensively, and if you do any Java Web development, you are familiar with this pattern. You can search through the code and find many references to a method called decorate. Decorating an object allows the user to customize and or chain the behavior to a specific need without impacting the original object. For example there is a decorate method on the TransformedMap object, which opens the door to many possibilities. it is worth noting that the Apache Commons Collections framework has changed recently so it is possible that something I'm writing about may have changed.

TransformedMap.decorate( )

Transformed Map is the ultimate transformer and I cover it a little more here. This class and method allows the user to manipulate the Key, Value, and or the backing object. So you could do something like this.

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;
import org.apache.commons.collections.Transformer;
import org.apache.commons.collections.map.TransformedMap;
import java.util.*;

public class DecoratingMaps {
    public static void main(String[] args) {
        List<g;tuple>  tuples = Arrays.asList( new tuple("a", "123"),
                                            new tuple("b", "456"),
                                            new tuple("c", "789"),
                                            new tuple("d", "012") );

        Map map = TransformedMap.decorate(new HashMap(), new Transformer() {
            public Object transform(Object o) {
                return ((tuple) o).getName();
            }
        }, new Transformer() {
            public Object transform(Object o) {
                return ((tuple) o).getValue();
            }
        });


        Iterator<tuple> iterator = tuples.iterator();
        while (iterator.hasNext()) {
            tuple tuple = iterator.next();
            map.put(tuple, tuple);
        }

        System.out.println(map.toString());
    }

    private static class tuple {
        String name;
        String value;

        private tuple(String name, String value) {
            this.name = name;
            this.value = value;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public String getValue() {
            return value;
        }

        public void setValue(String value) {
            this.value = value;
        }
    }
}

Which allows us to just Put an object in a map and let the map worry about what part is the key and which is the value. I've used this in the past and it allows me to ( with a factory ) build maps that know how to store the values alleviating the user from the burden of this knowledge, a paractice of seperation of concern. The code above produce a map like this

{d=012, b=456, c=789, a=123}

Now that I have shown you the transformed map class, what is a Lazy Map? A Lazy Map is simply a map that knows how to create or fetch a value based on the key. This is because the values may or may not be present in the map. So, there has to be some solid relationship between the value and key's uniqueness that it can calculate the value. The LRU or Least Recently Used map is typically a map with a static size that ages out content based on an algorithm for the purpose of keeping the object within a size limit ( or some other dimension such as time ). As I mentioned, this example uses Yahoo Stocks, so I abstracted the Yahoo commands into a Enum. The purpose of this enum is to bridge the gap between commands sent to the service and human readability. Here is the enum.


package com.blogspot.apachecommonstipsandtricks.lazymapexamples;

public enum YahooFinanaceCommands
{
    a("Ask"),
    b("Bid"),
    b4("Book Value"),
    c1("Change"),
    c8("After Hours Change (Real-time)"),
    d2("Trade Date"),
    e7("EPS Estimate Current Year"),
    f6("Float Shares"),
    j("52-week Low"),
    g3("Annualized Gain"),
    g6("Holdings Gain (Real-time)"),
    j1("Market Capitalization"),
    j5("Change From 52-week Low"),
    k2("Change Percent (Real-time)"),
    k5("Percebt Change From 52-week High"),
    l2("High Limit"),
    m2("Day's Range (Real-time)"),
    m5("Change From 200-day Moving Average"),
    m8("Percent Change From 50-day Moving Average"),
    o("Open"),
    p2("Change in Percent"),
    q("Ex-Dividend Date"),
    r2("P/E Ratio (Real-time)"),
    r7("Price/EPS Estimate Next Year"),
    s7("Short Ratio"),
    t7("Ticker Trend"),
    v1("Holdings Value"),
    w1("Day's Value Change"),
    y("Dividend Yield"),
    a2("Average Daily Volume"),
    b2("Ask (Real-time)"),
    b6("Bid Size"),
    c3("Commission"),
    d("Dividend/Share"),
    e("Earnings/Share"),
    e8("EPS Estimate Next Year"),
    g("Day's Low"),
    k("52-week High"),
    g4("Holdings Gain"),
    i("More Info"),
    j3("Market Cap (Real-time)"),
    j6("Percent Change From 52-week Low"),
    k3("Last Trade Size"),
    l("Last Trade (With Time)"),
    l3("Low Limit"),
    m3("50-day Moving Average"),
    m6("Percent Change From 200-day Moving Average"),
    n("Name"),
    p("Previous Close"),
    p5("Price/Sales"),
    r("P/E Ratio"),
    r5("PEG Ratio"),
    s("Symbol"),
    t1("Last Trade Time"),
    t8("1 yr Target Price"),
    v7("Holdings Value (Real-time)"),
    w4("Day's Value Change (Real-time)"),
    a5("Ask Size"),
    b3("Bid (Real-time)"),
    c("Change & Percent Change"),
    c6("Change (Real-time)"),
    d1("Last Trade Date"),
    e1("Error Indication (returned for symbol changed / invalid)"),
    e9("EPS Estimate Next Quarter"),
    h("Day's High"),
    g1("Holdings Gain Percent"),
    g5("Holdings Gain Percent (Real-time)"),
    i5("Order Book (Real-time)"),
    j4("EBITDA"),
    k1("Last Trade (Real-time) With Time"),
    k4("Change From 52-week High"),
    l1("Last Trade (Price Only)"),
    m("Day's Range"),
    m4("200-day Moving Average"),
    m7("Change From 50-day Moving Average"),
    n4("Notes"),
    p1("Price Paid"),
    p6("Price/Book"),
    r1("Dividend Pay Date"),
    r6("Price/EPS Estimate Current Year"),
    s1("Shares Owned"),
    t6("Trade Links"),
    v("Volume"),
    w("52-week Range"),
    x("Stock Exchange");

    private String description;

    YahooFinanaceCommands(String description)
    {
        this.description = description;
    }
}


Disclosure and credit needs to be in order here. First, I am not associated with Yahoo, NASDAQ, or NYSE in any form, so you are on your own with this and second I got all of these commands from the Website http://www.gummy-stuff.org/Yahoo-data.htm. Furthermore, Yahoo has a throttle on this service, and if you exceed it, you will be locked out for some time and even banned. You need to read the usage disclosures of Yahoo, NASDAQ, and NYSE.

So, as you can guess, I also needed to get a list of keys for testing and reference ( the Stock Symbols ). This was an daunting task, it took me hours of searching via Google before I found at least one exchange that had the symbols publicly accessible and downloadable. Again, a little disclosure, I am not associated with NASDAQ or the NYSE, please don't ask me for help. Although I got the NYSE working, I thought it easier to just demo NASDAQ and that is why there is code commented out. Before I get to deep, lets recap our objective. We need the keys for the NASDAQ, build a Lazy Map with a LRU of the stock values based on Yahoo stock service and some concurrency modification protection with the ConcurrentHashMap.

With the enum I can now assemble the URN of a URL for the purpose of pulling stocks from Yahoo and then based on the results, fill out a object called Stock. This is what the Stock object will look like.

package com.blogspot.apachecommonstipsandtricks.lazymapexamples.entity;

public class Stock implements Cloneable
{
    private String symbol;
    private String name;
    private Float fifetytwoWeekLow;
    private Float fifetytwoWeekHigh;
    private Float peRatioRealTime;
    private Float priceEPSEstimateNextYear;

    public Stock()
    {
    }

    public Stock(String symbol, String name, Float fifetytwoWeekLow, Float fifetytwoWeekHigh, Float peRatioRealTime,
                 Float priceEPSEstimateNextYear)
    {
        this.symbol = symbol;
        this.name = name;
        this.fifetytwoWeekLow = fifetytwoWeekLow;
        this.fifetytwoWeekHigh = fifetytwoWeekHigh;
        this.peRatioRealTime = peRatioRealTime;
        this.priceEPSEstimateNextYear = priceEPSEstimateNextYear;
    }

    public String getSymbol()
    {
        return symbol;
    }

    public void setSymbol(String symbol)
    {
        this.symbol = symbol;
    }

    public String getName()
    {
        return name;
    }

    public void setName(String name)
    {
        this.name = name;
    }

    public Float getFifetytwoWeekLow()
    {
        return fifetytwoWeekLow;
    }

    public void setFifetytwoWeekLow(Float fifetytwoWeekLow)
    {
        this.fifetytwoWeekLow = fifetytwoWeekLow;
    }

    public Float getFifetytwoWeekHigh()
    {
        return fifetytwoWeekHigh;
    }

    public void setFifetytwoWeekHigh(Float fifetytwoWeekHigh)
    {
        this.fifetytwoWeekHigh = fifetytwoWeekHigh;
    }

    public Float getPeRatioRealTime()
    {
        return peRatioRealTime;
    }

    public void setPeRatioRealTime(Float peRatioRealTime)
    {
        this.peRatioRealTime = peRatioRealTime;
    }

    public Float getPriceEPSEstimateNextYear()
    {
        return priceEPSEstimateNextYear;
    }

    public void setPriceEPSEstimateNextYear(Float priceEPSEstimateNextYear)
    {
        this.priceEPSEstimateNextYear = priceEPSEstimateNextYear;
    }

    @Override
    public Object clone() // throws CloneNotSupportedException
    {
        return new Stock(this.getSymbol(), this.getName(), this.getFifetytwoWeekLow(), this.getFifetytwoWeekHigh(),
                         this.getPeRatioRealTime(), this.getPriceEPSEstimateNextYear());
    }

    @Override
    public String toString()
    {
        return "Stock{" +
               "symbol='" + symbol + '\'' +
               ", name='" + name + '\'' +
               ", fifetytwoWeekLow=" + fifetytwoWeekLow +
               ", fifetytwoWeekHigh=" + fifetytwoWeekHigh +
               ", peRatioRealTime=" + peRatioRealTime +
               ", priceEPSEstimateNextYear=" + priceEPSEstimateNextYear +
               '}';
    }
}

So I will be requesting from the Yahoo Service a 52 week high and low value, the Price to Earning ratio ( Real Time ) and the Price EPS estimated next year. I will then put the results in the Stock object and store it in my LRU map.

Are you beginning to see a problem yet? If not, that is ok, let me explain. The LRU map disposes not just the value in the map, it also disposes both the value and the KEY.. pause .. the key too ..  pause .. So, if you wanted to get all keys via map.keySet() and then iterator through all the values, forget it. If your knee jerk reaction is to yell FOUL, think about it, it makes sense. The purpose is to reduce the memory foot print, iterating through the whole map completely undermines the LRU as it will have to go get everything and then throw out the results. But in our example, we know all the keys ( even if they change when a company is sold or changes it's name ). What to do.. gluing functionality together can be a real challenge, but fun, so lets get gluing!

So thinking out loud, we cant just use the LRU and Lazy Map, the final object will have to have some functionality beyond a map. Naturally a singleton utility class, we will call it StockFetcher. Stock Fetcher will both encapsulate the LRU / Lazy Map / Concurrent Hash Map, cache the Stock Symbols and age them out,  provides a get Stock method, and provide an iterator to go over the Stock symbols it knows about. Here it is.....

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;

import com.blogspot.apachecommonstipsandtricks.lazymapexamples.entity.Stock;
import org.apache.commons.collections.Transformer;
import org.apache.commons.collections.map.LRUMap;
import org.apache.commons.collections.map.LazyMap;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVStrategy;
import org.apache.commons.lang.StringUtils;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringReader;
import java.net.URL;

import java.util.concurrent.ConcurrentHashMap;
import java.util.*;

public class StockFetcher
{
    private static final int MaxCacheSize = 100;
    private static final long MilliSecondsToFlush = ( ( 1000L * 60L ) * 60L )* 24L;
    private static Timer timer = new Timer();

    // I broke these out, so you could easily modify the values without having to dig in the code.
    public static final YahooFinanaceCommands[] YahooCommands = new YahooFinanaceCommands[]{YahooFinanaceCommands.s,
            YahooFinanaceCommands.n,
            YahooFinanaceCommands.j,
            YahooFinanaceCommands.k,
            YahooFinanaceCommands.r2,
            YahooFinanaceCommands.r7};

    private static final Map<String, Stock> map = LazyMap.decorate(new ConcurrentHashMap(new LRUMap(MaxCacheSize)),
                                                                   new StockMarketFactoryTransformer());

    private static final List<String> KNOWN_STOCK_SYMBOLS = new ArrayList<String>();
    private static final String FTP_SYMBOLDIRECTORY_NASDAQ = "ftp://ftp.nasdaqtrader.com/symboldirectory/nasdaqlisted.txt";
    // private static final String FTP_SYMBOLDIRECTORY_NYSE= "http://www.nyse.com/attachment/activated_lrps.xls";
    // private static final String FTP_SYMBOLDIRECTORY_NYSE= "http://online.wsj.com/public/resources/documents/NYSE.csv";

    private static final StockFetcher ourInstance = new StockFetcher();

    /**
     * This is a singleton.
     */
    private StockFetcher()
    {
        fetchKeys();
    }

    /**
     * refer to singleton pattern
     * @return the static instance of this class.
     */
    public static StockFetcher getInstance()
    {
        return ourInstance;
    }

    /**
     * Method responsible for fetching all the NASDAQ Stocks symbols and setting up a timmer to do it again.
     */
    private static synchronized void fetchKeys()
    {
        timer.schedule(new KeyFlushTimerTask(), MilliSecondsToFlush);
        System.out.println("Fetching keys");
        int count = 0;
        try
        {
            URL url = new URL(FTP_SYMBOLDIRECTORY_NASDAQ);
            BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
            String str;
            // nice thing about doing the two in.readlines is it skips the header.
            if (((str = in.readLine()) != null))
            {
                while ((str = in.readLine()) != null)
                {
                    String[] strings = str.split("\\|");
                    if (strings.length == 6)
                    {
                        count++;
                        KNOWN_STOCK_SYMBOLS.add(strings[0]);
                    }
                }
            }
            in.close();
        }
        catch (IOException e)
        {
            throw new IllegalStateException("An exception occured while attempting to fetch the Stock quote.", e);
        }
        finally
        {
            System.out.println(" Read " + count + " symbols.");
        }
    }

    /**
     * Takes an Array of Yahoo Finanace comamnds and assembles the URL string that will be sent to Yahoo.
     * @param yfcs a vararg of YahooFinanaceCommands to send to yahoo.
     * @return a URL used to send to Yahoo.
     */
    private static String compileCommands(YahooFinanaceCommands... yfcs)
    {
        StringBuilder sb = new StringBuilder();
        if ( yfcs != null)
        {
            for (YahooFinanaceCommands yfc : yfcs)
            {
                sb.append(yfc.toString());
            }
        }
        if (sb.length() != 0)
        {
            return "&f=" + sb.toString();
        }
        return "";
    }

    /**
     * Builds a URN
     * @param symbols a var arg of symbols to fetch.
     * @return the urn
     */
    private static String query(String... symbols)
    {
        return StringUtils.join(symbols, "+");
    }

    /**
     * Get a stock based on it's nasdq symbol
     * @param symbol the nasdaq symbol
     * @return a stock object.
     */
    public Stock get(String symbol)
    {
        Stock aStock = map.get(symbol);
        return (aStock==null)?null:(Stock)aStock.clone();
    }

    /**
     * Iterates all the known stock symbols.
     * @return String interator.
     */
    public Iterator<String> iterator()
    {
        return ( new ArrayList<String>( KNOWN_STOCK_SYMBOLS ) ).iterator();
    }

    private static class StockMarketFactoryTransformer implements Transformer
    {
        public Object transform(Object o)
        {
            System.out.println("Fetching data for " + (String) o );
            Stock retStock = null;
            try
            {
                URL url = new URL("http", "finance.yahoo.com", 80,
                                  "/d/quotes.csv?s=" + query((String) o) + compileCommands(YahooCommands));
                BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
                String str;
                if (((str = in.readLine()) != null))
                {
                    do
                    {
                        CSVParser csvParser = new CSVParser(new StringReader(str),
                                                            CSVStrategy.DEFAULT_STRATEGY);

                        String[] strings = StringUtils.stripAll(csvParser.getLine());
                        // System.out.println(Arrays.toString(strings));
                        int i = 0;
                        retStock = new Stock();
                        retStock.setSymbol(strings[i++]);
                        retStock.setName(strings[i++]);
                        retStock.setFifetytwoWeekLow(safeFloatConvert(strings[i++]));
                        retStock.setFifetytwoWeekHigh(safeFloatConvert(strings[i++]));
                        retStock.setPeRatioRealTime(safeFloatConvert(strings[i++]));
                        retStock.setPriceEPSEstimateNextYear(safeFloatConvert(strings[i]));
                    }
                    while ((str = in.readLine()) != null);
                }
                in.close();
            }
            catch (IOException e)
            {
                throw new IllegalStateException("An exception occured while attempting to fetch the Stock quote.", e);
            }
            return retStock;
        }

        private Float safeFloatConvert(String string)
        {
            Float retFloat = null;
            if (!"N/A".equalsIgnoreCase(string))
            {
                try
                {
                    retFloat = Float.valueOf(string);
                }
                catch (NumberFormatException e)
                {
                    e.printStackTrace();
                }
            }
            return retFloat;
        }
    }
    
    private static class KeyFlushTimerTask extends TimerTask
    {
        public void run()
        {
            fetchKeys();
        }
    }
}

The variable MaxCacheSize allows us to set the maximum size of the LRU map ( up to 100 entities in this example ) while MilliSecondsToFlush is used by a java timer task ( see static Timer timer ) defined near the end of the Stock Fetcher class as KeyFlushTimerTask which extends TimerTask. This object will evict all the stock symbols in the KNOWN_STOCK_SYMBOLS ( not the map which you might want to do if it suits your needs ).

    private static final int MaxCacheSize = 100;
    private static final long MilliSecondsToFlush = ( ( 1000L * 60L ) * 60L )* 24L;
    private static Timer timer = new Timer();

So following those three lines is the following line..

    // I broke these out, so you could easily modify the values without having to dig in the code.
    public static final YahooFinanaceCommands[] YahooCommands = new YahooFinanaceCommands[]{YahooFinanaceCommands.s,
            YahooFinanaceCommands.n,
            YahooFinanaceCommands.j,
            YahooFinanaceCommands.k,
            YahooFinanaceCommands.r2,
            YahooFinanaceCommands.r7};

What I've done here is assembled a static array of YahooFinanaceCommands which will represent the command ( URN of the URL ) I will be using to call Yahoo with ( see the YahooFinanaceCommands enum above ).

So now onto the magic of the whole tutorial, the glue of the whole system. We will now decorate the map.

    private static final Map<String, Stock> map = LazyMap.decorate(
                                                            new ConcurrentHashMap(new LRUMap(MaxCacheSize)),
                                                                   new StockMarketFactoryTransformer());


What we have done so far is call the decorate method on the Apache Commons Lazy Map object backing it with a ConcurrentHashMap which in turn is backed by a LRU Map defined to the size of MaxCacheSize with a transformer called StockMarketFactoryTransformer. It is perfectly acceptable to Pump Your Fist and yell OH YEAH once this sinks in! So our new map will take on the characteristics of three different Map Objects ( LRUMap, ConcurrentHashMap and LRUMap ). Chaining them all together to create a whole new kind of Map. A Map, that behaves exactly as I want it to again:

Pump Fist and yell OH YEAH!

Special Note here regarding ConcurrentHashMap and maps in general. Naturally maps are not thread safe and using a ConcurrentHashMap will impact the performance, but for our purpose this is all acceptable. It might not be in your environment so give it some consideration before you copy-cut and paste this code.

Moving on, you will have seen KNOWN_STOCK_SYMBOLS, FTP_SYMBOLDIRECTORY_NASDAQ, and the commented out URLs called FTP_SYMBOLDIRECTORY_NYSE. There is no more hiding it; the problem with this code/pattern involves knowing what the keys are. In an attempt to capture the keys (which are the stock symbols), I will hold the stock symbols in FTP_SYMBOLDIRECTORY_NASDAQ and since this is a singleton, it will happen once in the private constructor when fetchKeys is fired off ( and by the way, this is when the timer task schedules itself ). I know this solution is not perfect, eventually the Stock market keys will be out of sync as companies come and go. Not to mention the problems and complexity associated with the file at FTP_SYMBOLDIRECTORY_NASDAQ, but I never made a claim that this code is perfect. In most cases, it is just good enough. So to re-iterate, the singleton is fired up, the function fetchKeys is fired which will fetch all the symbols in the file stored on the NASDAQ server and adding all of them to the array KNOWN_STOCK_SYMBOLS and scheduling the timer to fire the method again. I have exposed the keys via an iterator method further down, but as I mentioned earlier this defeats the whole purpose of the LRU Map if you iterate over them.

The function compileCommands takes a Java 1.5 varargs of YahooCommands to build part of the URN to the URL of the Yahoo Server. I did this so, you can add more values if you desire. This function will be used later when we pull the stock value. The values are parsed then stored in the Stock object and placed in the Lazy Map / ConcurrentHashMap / LRUMap ( shweeeet ). All of this is done by the inner class StockMarketFactoryTransformer. This object implements transformer and was injected in the LazyMap Decorate method.

    private static class StockMarketFactoryTransformer implements Transformer
    {
        public Object transform(Object o)
        {
            System.out.println("Fetching data");
            Stock retStock = null;
            try
            {
                URL url = new URL("http", "finance.yahoo.com", 80,
                                  "/d/quotes.csv?s=" + query((String) o) + compileCommands(YahooCommands));
                BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
                String str;
                if (((str = in.readLine()) != null))
                {
                    do
                    {

                        CSVParser csvParser = new CSVParser(new StringReader(str),
                                                            CSVStrategy.DEFAULT_STRATEGY);

                        String[] strings = StringUtils.stripAll(csvParser.getLine());
                        // System.out.println(Arrays.toString(strings));
                        int i = 0;
                        retStock = new Stock();
                        retStock.setSymbol(strings[i++]);
                        retStock.setName(strings[i++]);
                        retStock.setFifetytwoWeekLow(safeFloatConvert(strings[i++]));
                        retStock.setFifetytwoWeekHigh(safeFloatConvert(strings[i++]));
                        retStock.setPeRatioRealTime(safeFloatConvert(strings[i++]));
                        retStock.setPriceEPSEstimateNextYear(safeFloatConvert(strings[i]));
                    }
                    while ((str = in.readLine()) != null);
                }
                in.close();
            }
            catch (IOException e)
            {
                throw new IllegalStateException("An exception occured while attempting to fetch the Stock quote.", e);
            }
            return retStock;
        }

        private Float safeFloatConvert(String string)
        {
            Float retFloat = null;
            if (!"N/A".equalsIgnoreCase(string))
            {

                try
                {
                    retFloat = Float.valueOf(string);
                }
                catch (NumberFormatException e)
                {
                    e.printStackTrace();
                }
            }
            return retFloat;
        }
    }


So here is a basic example of it's usage and honestly not a good example. In this example, we are in a infinite loop, the first time we instantiate the StockFetcher via getInstance it loads all the keys ( stock symbols from nasdaq ). Then we call get AAPL and since it is not in the cache the transformer executes and pulls the stock and stores it. Further we sleep for a few seconds, and then attempt to get the stock XXIA, again it doesn't exist so it goes and gets it... while in the loop we make the same calls to AAPL and XXIA and since they are in the cache, we hit the cache. Eventually the stocks are evicted from the map if more than 100 unique hits occur on the map.

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;

import com.blogspot.apachecommonstipsandtricks.lazymapexamples.entity.Stock;


public class StockTickerTest
{
    public static void main(String[] args)
    {
        do{
            try
            {
                Stock stock = StockFetcher.getInstance().get("AAPL");
                System.out.println("stock = " + stock);
                System.out.println("going to sleep.");
                Thread.sleep(10000);
                stock = StockFetcher.getInstance().get("XXIA");
                System.out.println("stock = " + stock);
                System.out.println("going to sleep.");
                Thread.sleep(10000);
            }
            catch (InterruptedException e)
            {
                e.printStackTrace();
            }

        }while(true);

    }

}


I hope you enjoyed this and found it helpful. Please feel free to send me comments or suggestions.
Author: Philip Senger

Sunday, February 7, 2010

The Perls and Perils of BeanUtils copyProperties

I’m still working on my Lazy Map example and it’s been a long time since I posted so I decided I should write a quick small post. I decided to talk about BeanUtils.copyProperties and how it can be misused.

When you need to copy a large number of members from one Data Transfer Object or Value Object (DTO/VO) to another, copyProperties is indeed a powerful and useful tool. However as the adage goes ”with great power comes a great responsibility”, this utility has a dark side. In my opinion it’s a terrible side effect that is often misunderstood by the average developer, hence the title of this article.

So, what does it do and why would I be interested in using it anyway? Well, as the name implies, it simple copies the members from one bean to another, matching the getter and setter names to do so. It only has three Run Time Exceptions. These exceptions are IllegalAccessException, IllegalArgumentException, and InvocationTargetException.

IllegalAccessException - if the caller does not have access to the property accessor method
IllegalArgumentException - if the dest or orig argument is null or if the dest property type is different from the source type and the relevant converter has not been registered.
InvocationTargetException - if the property accessor method throws an exception
Potentially, It saves time by allowing a developer to write one line of code versus 10 or even 20 lines of code.

You might be asking yourself “How could something so good have any kind of side effect that would be considered bad?” I personally have found that this class has got to be one of the biggest misused utilities I have ever seen. Copying beans, when there is no contract such as an Interface or an Abstract class involved usually means you might get something you didn’t expect. An Anonymous User hinted at the problem in one of his comments… Refactoring. Say the DTO/VO’s you are going to copy have similar member names, but no contract and you copy them… several months later someone is working on code in some part of the program and doesn’t like the name so changes it. Most IDEA such as Intellij or Eclipse provide this feature, but in their defense, there is just no way to know that the code is now broken. If there was a interface or Abstract class it would have corrected it.

Here is an example of a failure. The following two DTO/VO objects dont exacly match because someone deleted a member. The test catches this error, but as you can see, the test really isnt a good test. For this example, I will be using beanutils 1.8.2, collections 3.2.1, lang 2.4, logging 1.1.1, and junit 4.8.1

import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;
import org.apache.commons.lang.builder.ToStringBuilder;

public class ValueObjectA {
private String firstname;
private String lastname;
private String cv;
private String linkedin;
private String blog;
private int age;
private boolean hacker;

public ValueObjectA() { }

public ValueObjectA(String firstname, String lastname, String cv, String linkedin, String blog, int age, boolean hacker) {
this.firstname = firstname;
this.lastname = lastname;
this.cv = cv;
this.linkedin = linkedin;
this.blog = blog;
this.age = age;
this.hacker = hacker;
}

public String getFirstname() {
return firstname;
}

public void setFirstname(String firstname) {
this.firstname = firstname;
}

public String getLastname() {
return lastname;
}

public void setLastname(String lastname) {
this.lastname = lastname;
}

public String getCv() {
return cv;
}

public void setCv(String cv) {
this.cv = cv;
}

public String getLinkedin() {
return linkedin;
}

public void setLinkedin(String linkedin) {
this.linkedin = linkedin;
}

public String getBlog() {
return blog;
}

public void setBlog(String blog) {
this.blog = blog;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public boolean isHacker() {
return hacker;
}

public void setHacker(boolean hacker) {
this.hacker = hacker;
}

@Override
public boolean equals(Object o) {
return EqualsBuilder.reflectionEquals(this, o);

}

@Override
public int hashCode() {
return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public String toString() {
return ToStringBuilder.reflectionToString(this);
}
}


import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;
import org.apache.commons.lang.builder.ToStringBuilder;

public class ValueObjectB {
private String firstname;
private String lastname;
private String cv;
private String linkedin;
private int age;
private boolean hacker;

public ValueObjectB() { }

public ValueObjectB(String firstname, String lastname, String cv, String linkedin, int age, boolean hacker) {
this.firstname = firstname;
this.lastname = lastname;
this.cv = cv;
this.linkedin = linkedin;
this.age = age;
this.hacker = hacker;
}

public String getFirstname() {
return firstname;
}

public void setFirstname(String firstname) {
this.firstname = firstname;
}

public String getLastname() {
return lastname;
}

public void setLastname(String lastname) {
this.lastname = lastname;
}

public String getCv() {
return cv;
}

public void setCv(String cv) {
this.cv = cv;
}

public String getLinkedin() {
return linkedin;
}

public void setLinkedin(String linkedin) {
this.linkedin = linkedin;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public boolean isHacker() {
return hacker;
}

public void setHacker(boolean hacker) {
this.hacker = hacker;
}

@Override
public boolean equals(Object o) {
return EqualsBuilder.reflectionEquals(this, o);

}

@Override
public int hashCode() {
return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public String toString() {
return ToStringBuilder.reflectionToString(this);
}    
}


So, if for some reason you could not make you're DTO/VO's implement an interface or extend an abstract class, you could set up a test like this:

import org.apache.commons.beanutils.BeanUtils;
import org.apache.commons.collections.CollectionUtils;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;
import org.junit.Test;
import static org.junit.matchers.JUnitMatchers.hasItem;

import java.lang.reflect.InvocationTargetException;
import java.util.Collection;
import java.util.Map;

public class BeanUtilsExample {

@Test
public void myTest() throws InvocationTargetException, IllegalAccessException, NoSuchMethodException {

ValueObjectA userA = new ValueObjectA("Philip", "Senger", "http://www.visualcv.com/philipsenger", "http://www.linkedin.com/in/philipsenger", "http://www.apachecommonstipsandtricks.blogspot.com/", 42, true);
ValueObjectB userB = new ValueObjectB();
BeanUtils.copyProperties(userB, userA);

Map<String, String> a = BeanUtils.describe(userA);
Map<String, String> b = BeanUtils.describe(userB);

for (String keyB : b.keySet()) {
assertThat(a.keySet(), hasItem(keyB));
}

for (String keyA : a.keySet()) {
assertThat(b.keySet(), hasItem(keyA));
}

// or you could test this way.

Collection disjunction = CollectionUtils.disjunction(a.keySet(), b.keySet());
assertThat( disjunction.size(), is(0));

}
}


If you didn't want to set yourself up for failure or couldn't include a unit test like the one above, here is one way to change the two DTO/VO's.

public interface ValueObject {
String getFirstname();

void setFirstname(String firstname);

String getLastname();

void setLastname(String lastname);

String getCv();

void setCv(String cv);

String getLinkedin();

void setLinkedin(String linkedin);

String getBlog();

void setBlog(String blog);

int getAge();

void setAge(int age);

boolean isHacker();

void setHacker(boolean hacker);
}

import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;
import org.apache.commons.lang.builder.ToStringBuilder;

public class ValueObjectA implements ValueObject {
private String firstname;
private String lastname;
private String cv;
private String linkedin;
private String blog;
private int age;
private boolean hacker;

public ValueObjectA() {
}

public ValueObjectA(String firstname, String lastname, String cv, String linkedin, String blog, int age, boolean hacker) {
this.firstname = firstname;
this.lastname = lastname;
this.cv = cv;
this.linkedin = linkedin;
this.blog = blog;
this.age = age;
this.hacker = hacker;
}

public String getFirstname() {
return firstname;
}

public void setFirstname(String firstname) {
this.firstname = firstname;
}

public String getLastname() {
return lastname;
}

public void setLastname(String lastname) {
this.lastname = lastname;
}

public String getCv() {
return cv;
}

public void setCv(String cv) {
this.cv = cv;
}

public String getLinkedin() {
return linkedin;
}

public void setLinkedin(String linkedin) {
this.linkedin = linkedin;
}

public String getBlog() {
return blog;
}

public void setBlog(String blog) {
this.blog = blog;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public boolean isHacker() {
return hacker;
}

public void setHacker(boolean hacker) {
this.hacker = hacker;
}

@Override
public boolean equals(Object o) {
return EqualsBuilder.reflectionEquals(this, o);

}

@Override
public int hashCode() {
return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public String toString() {
return ToStringBuilder.reflectionToString(this);
}
}

import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;
import org.apache.commons.lang.builder.ToStringBuilder;

public class ValueObjectB implements ValueObject {
private String firstname;
private String lastname;
private String cv;
private String linkedin;
private String blog;
private int age;
private boolean hacker;

public ValueObjectB() { }

public ValueObjectB(String firstname, String lastname, String cv, String linkedin, String blog, int age, boolean hacker) {
this.firstname = firstname;
this.lastname = lastname;
this.cv = cv;
this.linkedin = linkedin;
this.blog = blog;
this.age = age;
this.hacker = hacker;
}

public String getFirstname() {
return firstname;
}

public void setFirstname(String firstname) {
this.firstname = firstname;
}

public String getLastname() {
return lastname;
}

public void setLastname(String lastname) {
this.lastname = lastname;
}

public String getCv() {
return cv;
}

public void setCv(String cv) {
this.cv = cv;
}

public String getLinkedin() {
return linkedin;
}

public void setLinkedin(String linkedin) {
this.linkedin = linkedin;
}

public String getBlog() {
return blog;
}

public void setBlog(String blog) {
this.blog = blog;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public boolean isHacker() {
return hacker;
}

public void setHacker(boolean hacker) {
this.hacker = hacker;
}

@Override
public boolean equals(Object o) {
return EqualsBuilder.reflectionEquals(this, o);

}

@Override
public int hashCode() {
return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public String toString() {
return ToStringBuilder.reflectionToString(this);
}    
}


In conclusion, this command is powerful, but if you use it, I advise you to consider using a Interface or an Abstract class. Hope you enjoy this post, and feel free to drop me an email.

Author: Philip A Senger

Tuesday, March 31, 2009

Examples of Lazy Lists and Factories

In this post, I'm going to examine Lazy Lists and the Apache Factory.

In accordance with most of the Apache Commons Collection utilities, a LazyList is simply a decorator (and at this point that shouldn't be a surprise). It decorates another List creating a sort-of "list on demand" list. Calling the method get(int index) when the index is greater than the size of the list will grow the list with objects from the given factory. Using the get(int index) method to get an object that the factory did not fill in will result in an IndexOutOfBoundsException.

For all of the examples, lets use the following simple factory.
The Code:

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class LazyListFactory implements Factory
{
private static int i = 0;
private List<String> strings = new ArrayList();
public LazyListFactory(List<String> strings)
{
this.strings.addAll(strings);
}
public Object create()
{
return this.strings.get(i++);
}
}

In this example Java code we have implemented the Factory interface, which defines a Object create(), but more importantly the constructor takes as an argument a List of Strings. These strings are effectively the "feed" that the create method will use. A more practical solution would be some form of a data source, but that would result in a lot of code, and this is simple. So with this factory, let us create some lazy data!

Our first java example of a lazy list.
The Code:

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;
import java.util.*;
import org.apache.commons.collections.list.*;
public class ListManager
{
public static void main(String[] args)
{
// First lets create a simple list "A" through "C"
List<String> backingList = new ArrayList<String>( Arrays.asList("A","B","C") );
// Now we create a Decorated List with homegrown class called LazyListFactory which implements Factory
List<String> list = LazyList.decorate( GrowthList.decorate( backingList ), new LazyListFactory( Arrays.asList("D","E","F","G") ) );
System.out.println("The size method on list thinks there are " + list.size() + " elements in the array list. but Im going to get 7!");
// as the more and more elements are requested, the factory creates more..
for (int i = 0; i < 7; i++)
{
Object o = list.get(i);
System.out.println( " Element [" + i + "] = " + o );
}
}
}


Results in

The size method on list thinks there are 3 elements in the array list. but Im going to get 7!
Element [0] = A
Element [1] = B
Element [2] = C
Element [3] = D
Element [4] = E
Element [5] = F
Element [6] = G


In this example, we first created a list, with Strings "A","B", and "C" than we decorated it, and attached a factory constructed with four more additional strings "D" through "G". I knew there was seven items in the list, so I forced it to loop to seven. Did you notice, the list thought the size was only 3. You might be thinking, which I am too, that’s not very helpful. How can I tell how much to loop through? Let’s look at this same example, but with some minor modifications.

The Code:

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;
import java.util.*;
import org.apache.commons.collections.list.*;
public class ListManager
{
public static void main(String[] args)
{
// First lets create a simple list "A" through "C"
List<String> backingList = new ArrayList<String>( Arrays.asList("A","B","C") );
// Now we create a Decorated List with homegrown class called LazyListFactory which implements Factory
List<String> list = LazyList.decorate( GrowthList.decorate( backingList ), new LazyListFactory( Arrays.asList("D","E","F","G") ) );

// Now puit "Z" in position 15
list.set( 15, "Z" );

System.out.println("After putting \"Z\" in position 15, the size is equal to " + list.size());

System.out.println("Which allows us to us a standard for-loop with a IndexOutOfBoundsException try-catch block.");
for (int i = 0; i < list.size(); i++)
{
try
{
Object o = list.get(i);
System.out.println( " Element [" + i + "] = " + o );
} catch (IndexOutOfBoundsException e) {}
}

}
}


You might have immediately noticed I put another String, "Z" at position 15, and now, the size is 15. I find it likely that this might be a solution to the "unknown size problem". The problem statement might go something like this, create a LazyList and set the last element to a known end state. Therefore, the user could pick what they wanted to consume out of the list while knowing the end is near at 15.

The results of the program above would look like this.

After putting "Z" in position 15, the size is equal to 16
Which allows us to us a standard for-loop with a IndexOutOfBoundsException try-catch block.
Element [0] = A
Element [1] = B
Element [2] = C
Element [3] = D
Element [4] = E
Element [5] = F
Element [6] = G
Element [15] = Z

If you are hung up on swallowing the IndexOutOfBoundsException exception or simply don’t like the idea of having to put in the last element this next solution may suite you. It uses the Java 1.5 for-loop.

The Code:

package com.blogspot.apachecommonstipsandtricks.lazymapexamples;
import java.util.*;
import org.apache.commons.collections.list.*;
public class ListManager
{
public static void main(String[] args)
{
// First lets create a simple list "A" through "C"
List<String> backingList = new ArrayList<String>( Arrays.asList("A","B","C") );
// Now we create a Decorated List with homegrown class called LazyListFactory which implements Factory
List<String> list = LazyList.decorate( GrowthList.decorate( backingList ), new LazyListFactory( Arrays.asList("D","E","F","G") ) );

// Now puit "Z" in position 15
list.set( 15, "Z" );

System.out.println("With a Java 1.5 for-loop the results look like this." );
int i = 0;
for (String s : list)
{
System.out.println( " Element [" + i++ + "] = " + s );
}
}
}


Like every solution, there is a draw back to this method, when an element does not exist, the result is null. In this example, I print every element, including the nulls. Let's examine the output.

With a Java 1.5 for-loop the results look like this.
Element [0] = A
Element [1] = B
Element [2] = C
Element [3] = null
Element [4] = null
Element [5] = null
Element [6] = null
Element [7] = null
Element [8] = null
Element [9] = null
Element [10] = null
Element [11] = null
Element [12] = null
Element [13] = null
Element [14] = null
Element [15] = Z


Each solution I have covered has some form of a draw back. I hope you can find one suitable for you. I will be examining the LazyMap in my next installment; it is well suited for hierarchical data.


Thanks for reading this article, I look forward to any questions or feedback. Please drop me a line if you want me to cover something in particular or would like to point something useful out.
Author: Philip A Senger

Tuesday, February 17, 2009

Examples of Bag and MultiKeys

In this installment, I’m going to discuss Java usages tips and tricks for the Apache Commons Collection Bag interface. Honestly, the first time I saw the Apache Commons Collection Bags API, I thought "Well, that isn't very useful." The Bag interface states the intention of the class; it is to count occurrences of objects. Like all the products in the Apache Commons Collections, Predicates, Closures, and Transformers can be plugged into the bag at run time to determine the behavior. I ran some tests, which are at the end of the article. The cost of using the framework is negligible, at about 10 to 20 milliseconds. This of course is compared to a traditional method of sampling counting occurrences and stuffing into a map. You can be the judge of the test, let me know if you think the test is accurate.

So, what kind of problem justifies adding the complexity of Bags to a project? Joining the magic of a Transformer, MultiKey, and a decorator you can produce a dynamic solution for counting groups of occurrences of properties of beans in a collection. This would be suitable for a generic controller responsible for counting groups of properties. It would be very helpful in a reporting system. Let’s take a look at the sample and walk through the code.

The Code:

package com.blogspot.apachecommonstipsandtricks.bags;
import java.util.*;
import java.net.*;
import java.io.*;
import java.lang.reflect.*;
import org.apache.commons.collections.*;
import org.apache.commons.collections.comparators.*;
import org.apache.commons.collections.keyvalue.*;
import org.apache.commons.collections.bag.*;
import org.apache.commons.collections.bag.HashBag;
import org.apache.commons.lang.*;
import org.apache.commons.beanutils.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class MagicBagOfTricks
{
public static void main(String[] args) throws IOException, InvocationTargetException, NoSuchMethodException, IllegalAccessException
{
List<LaborForce> list = new ArrayList<LaborForce>();
URL url = new URL("http", "1796193846474123283-a-1802744773732722657-s-sites.googlegroups.com", 80, "/site/psenger/Home/data.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
if (((str = in.readLine()) != null))
{
do
{
String[] strings = str.split("\t");
strings = StringUtils.stripAll(strings);

State state = State.valueOf(StringUtils.trim(strings[0]));
Gender gender = "Male".equals(StringUtils.trim(strings[1])) ? Gender.Male : Gender.Female;
Integer year = Integer.valueOf(StringUtils.trim(strings[2]));

list.add(new LaborForce(state, gender, year));
} while ((str = in.readLine()) != null);
}
in.close();

Map map = PropertyUtils.describe(new LaborForce());
Set s = map.keySet();
s.remove("class");
System.out.println("All the possible properties on the LaborForce bean are [" + StringUtils.join(s.toArray(),",") + "]");

Bag masterBag = new HashBag();
Bag genderBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender" } ) );
Bag genderYearBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" } ) );
Bag genderYearStateBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" , "state" } ) );

genderBag.addAll( list );
genderYearBag.addAll( list );
genderYearStateBag.addAll( list );

Comparator comparator = ComparatorUtils.chainedComparator(new Comparator[]{new MultiKeyCompartor(0),new MultiKeyCompartor(1),new MultiKeyCompartor(2)});
Set<MultiKey> set = new TreeSet<MultiKey>(comparator);
set.addAll(masterBag.uniqueSet());
for (MultiKey multiKey : set)
{
System.out.println( "[" +StringUtils.join(multiKey.getKeys(),',') + "] = " + masterBag.getCount(multiKey));
}
}
private static class PropertiesMultiKeyTransformer implements Transformer
{
String[] methodNames;
private PropertiesMultiKeyTransformer(String[] methodNames)
{
this.methodNames = methodNames;
}
public Object transform(Object o)
{
List<Object> ooos = new ArrayList<Object>();
for (String methodName : methodNames)
{
try
{
ooos.add(PropertyUtils.getProperty(o, methodName));
}
catch (Exception e)
{
throw new FunctorException(e);
}
}
return new MultiKey(ooos.toArray(new Object[ooos.size()]));
}
}
private static class MultiKeyCompartor implements Comparator<MultiKey>
{
private int i;
private MultiKeyCompartor(int i)
{
this.i = i;
}
public int compare(MultiKey o1, MultiKey o2)
{
Object[] keys1 = o1.getKeys();
Object[] keys2 = o2.getKeys();
Object oo1 = null;
try
{
oo1 = keys1[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
Object oo2 = null;
try
{
oo2 = keys2[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
NullComparator nullComparator = new NullComparator(false);
return nullComparator.compare(oo1, oo2);
}
}
}


Between lines 18 and 36 of the MagicBagOfTricks class, we fetch the sample data. The data is in the format of StatetabGendertabyear. If you want to download the data, you can get it here.


List<LaborForce> list = new ArrayList<LaborForce>();
URL url = new URL("http", "1796193846474123283-a-1802744773732722657-s-sites.googlegroups.com", 80, "/site/psenger/Home/data.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
if (((str = in.readLine()) != null))
{
do
{
String[] strings = str.split("\t");
strings = StringUtils.stripAll(strings);

State state = State.valueOf(StringUtils.trim(strings[0]));
Gender gender = "Male".equals(StringUtils.trim(strings[1])) ? Gender.Male : Gender.Female;
Integer year = Integer.valueOf(StringUtils.trim(strings[2]));

list.add(new LaborForce(state, gender, year));
} while ((str = in.readLine()) != null);
}
in.close();


The purpose of code on line 38 through 41 is to just print the properties of the bean, LaborForce, minus the class method. It is important to see how the PropertyUtils describes the bean, because we will be accessing it later via the same API.


Map map = PropertyUtils.describe(new LaborForce());
Set s = map.keySet();
s.remove("class");
System.out.println("All the possible properties on the LaborForce bean are [" + StringUtils.join(s.toArray(),",") + "]");


On line 43, we create a single HashBag object. I will refer to this HashBag object as the masterBag. A HashBag is an implementation of Bag with the characteristics of a HashSet. When objects are stored in the HashBag, the inserted object’s hash code is assessed, just like a HashSet, and ensures the uniqueness of the object in the Bag as it is stored. You might think the count would be 1 no mater how many times you insert the object, but in fact, the bag counts the inserted and removed occurrences. Even more confusing is the fact that there is an iterator method that returns all multiple occurrences and a uniqueSet method that pulls a set of objects. Line 44 through 46, is where the magic happens. We create three decorated Bags, with TransformedBag.decorate. All of them with variations of the PropertiesMultiKeyTransformer and backed by a single HashBag, the masterBag from line 43. When an object is inserted into one of the decorated HashBags, it is transformed by PropertiesMultiKeyTransformer and the results are physically placed in the masterBag. PropertiesMultiKeyTransformer uses the PropertyUtils to pull all the given properties off the bean to create a MultiKey object. It’s important that the we always put the same property in each object index in the array of objects of the MultiKey. If we didn’t, we would get an exception. Furthermore, we can’t transform the object into an array of objects because HashCode doesn’t exist on an Array of Objects. This becomes a problem when we insert Objects into the bag. The MultiKey is a perfect fit for this problem and that is the reason for its use. This class is effectively a wrapper class.


Bag masterBag = new HashBag();
Bag genderBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender" } ) );
Bag genderYearBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" } ) );
Bag genderYearStateBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" , "state" } ) );


On line 48 through 50 we add the object to the decorated transformed bags, which are all backed by the masterBag.


genderBag.addAll( list );
genderYearBag.addAll( list );
genderYearStateBag.addAll( list );


On line 52 through 58 I ask the masterBag for a set, load the results in a TreeSet that has a chained Transformer. The results are a sorted set of MultiKeys with counts of the occurrences of the vectors in the list.


Comparator comparator = ComparatorUtils.chainedComparator(new Comparator[]{new MultiKeyCompartor(0),new MultiKeyCompartor(1),new MultiKeyCompartor(2)});
Set<MultiKey> set = new TreeSet<MultiKey>(comparator);
set.addAll(masterBag.uniqueSet());
for (MultiKey multiKey : set)
{
System.out.println( "[" +StringUtils.join(multiKey.getKeys(),',') + "] = " + masterBag.getCount(multiKey));
}


The inner class PropertiesMultiKeyTransformer is a Transformer constructed with an array of Strings that represent the method names to use with PropertyUtils.getProperty on the beans in the list. It dynamically pulls the given properties from the bean and loads them into the new MultiKey. There is a limit to the number of Objects a MultiKey can accept, but for this demo, we are ok. In addition, the Objects in the MultiKey have to be consistently in the same order for each comparator. We are using a ComparatorUtils.chainedComparator of MultiKeyCompartor which will throw a ClassCastException if the objects are the wrong order.


private static class PropertiesMultiKeyTransformer implements Transformer
{
String[] methodNames;
private PropertiesMultiKeyTransformer(String[] methodNames)
{
this.methodNames = methodNames;
}
public Object transform(Object o)
{
List<Object> ooos = new ArrayList<Object>();
for (String methodName : methodNames)
{
try
{
ooos.add(PropertyUtils.getProperty(o, methodName));
}
catch (Exception e)
{
throw new FunctorException(e);
}
}
return new MultiKey(ooos.toArray(new Object[ooos.size()]));
}
}
private static class MultiKeyCompartor implements Comparator<MultiKey>
{
private int i;
private MultiKeyCompartor(int i)
{
this.i = i;
}
public int compare(MultiKey o1, MultiKey o2)
{
Object[] keys1 = o1.getKeys();
Object[] keys2 = o2.getKeys();
Object oo1 = null;
try
{
oo1 = keys1[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
Object oo2 = null;
try
{
oo2 = keys2[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
NullComparator nullComparator = new NullComparator(false);
return nullComparator.compare(oo1, oo2);
}
}


The code for the data transfer object used within this project is called the LaborForce object. See the following sample for the object. You can find State and Gender in some of the other projects used in this blog.

The Code:

package com.blogspot.apachecommonstipsandtricks.bags;
import com.blogspot.apachecommonstipsandtricks.*;
public class LaborForce
{
private State state;
private Gender gender;
private int year;
public LaborForce()
{
}
public LaborForce(State state, Gender gender, int year)
{
this.state = state;
this.gender = gender;
this.year = year;
}
public State getState()
{
return state;
}
public void setState(State state)
{
this.state = state;
}
public Gender getGender()
{
return gender;
}
public void setGender(Gender gender)
{
this.gender = gender;
}
public int getYear()
{
return year;
}
public void setYear(int year)
{
this.year = year;
}
}


The results:

All the possible properties on the LaborForce bean are [state,gender,year]
[Male] = 7842
[Male,1950] = 134
[Male,1950,AL] = 1
[Male,1950,AK] = 6
[Male,1950,AZ] = 2
[Male,1950,CA] = 1
[Male,1950,CO] = 4
[Male,1950,CT] = 3
[Male,1950,DE] = 3
[Male,1950,DC] = 1
[Male,1950,FL] = 2
[Male,1950,GA] = 2
[Male,1950,HI] = 4
..
..
[Female,2008,SD] = 3
[Female,2008,TN] = 2
[Female,2008,TX] = 6
[Female,2008,UT] = 2
[Female,2008,VT] = 1
[Female,2008,VA] = 2
[Female,2008,WA] = 3
[Female,2008,WI] = 3


Performance Test


The test I ran was very simple. I wrapped a timer around parts of the code that were unique to the process. And the results of using the Bag are small, compared to a traditional method.

The code:

package com.blogspot.apachecommonstipsandtricks.bags;
import java.util.*;
import java.text.*;
import java.net.*;
import java.io.*;
import org.apache.commons.collections.*;
import org.apache.commons.collections.bag.*;
import org.apache.commons.collections.bag.HashBag;
import org.apache.commons.lang.time.*;
import org.apache.commons.lang.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class BagStressTest
{
private static final int NumberOfTimesToRun = 1000;
private static final int MasterListSize = 10;

public static void main(String[] args) throws IOException
{
int i = 1;
SimpleDateFormat sdf = new SimpleDateFormat("ss.SSS");
StopWatch sw;
Bag bag = null;
Map<State, Integer> map = null;
long sumOfTime = 0;

List<LaborForce> list = new ArrayList<LaborForce>();
URL url = new URL("http", "1796193846474123283-a-1802744773732722657-s-sites.googlegroups.com", 80, "/site/psenger/Home/data.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
if (((str = in.readLine()) != null))
{
do
{
String[] strings = str.split("\t");
strings = StringUtils.stripAll(strings);

State state = State.valueOf(StringUtils.trim(strings[0]));
Gender gender = "Male".equals(StringUtils.trim(strings[1])) ? Gender.Male : Gender.Female;
Integer year = Integer.valueOf(StringUtils.trim(strings[2]));

list.add(new LaborForce(state, gender, year));
} while ((str = in.readLine()) != null);
}
in.close();

List<LaborForce> listOfSampleData = new ArrayList<LaborForce>();
for (int j = 0; j < MasterListSize; j++)
{
listOfSampleData.addAll( list );
}
System.out.println("The sample listOfSampleData length is " + listOfSampleData.size());
System.out.println(" ---------- Start Test ---------- ");
sumOfTime = 0;
sw = new StopWatch();
for (int j = 0; j < NumberOfTimesToRun; j++)
{
sw.start();
bag = TransformedBag.decorate(new HashBag(), new Transformer()
{
public Object transform(Object o)
{
LaborForce dto = (LaborForce) o;
return dto.getState();
}
});
bag.addAll(listOfSampleData);
sw.stop();
sumOfTime += sw.getTime();
sw.reset();
}
System.out.println("With a Bag, Average Time in seconds and milliseconds is " + sdf.format( new Date( sumOfTime / NumberOfTimesToRun ) ) );
System.out.println(" ---------- End Test ---------- ");

// Alternative Way.
System.out.println(" ---------- Start Test ---------- ");
sumOfTime = 0;
sw.reset();
for (int j = 0; j < NumberOfTimesToRun; j++)
{
sw.start();
map = new HashMap<State, Integer>();
for (LaborForce lbf : listOfSampleData)
{
Integer count = map.get(lbf.getState());
if (null == count)
{
count = 0;
}
count++;
map.put(lbf.getState(), count);
}
sw.stop();
sumOfTime += sw.getTime();
sw.reset();
}
System.out.println("Traditional counting, Average Time in seconds and milliseconds is " + sdf.format( new Date( sumOfTime / NumberOfTimesToRun ) ) );
System.out.println(" ---------- End Test ---------- ");
}
}


The results:

The sample listOfSampleData length is 156000
---------- Start Test ----------
With a Bag, Average Time in seconds and milliseconds is 00.032
---------- End Test ----------
---------- Start Test ----------
Traditional counting, Average Time in seconds and milliseconds is 00.026
---------- End Test ----------


Thanks for reading this article, I look forward to any questions or feedback.

Author: Philip A Senger

Friday, February 13, 2009

Examples of Set Theory in Java with Apache Commons Collections

Before I get started on Apache Commons Collections Unions, intersections, and sub collections let me clarify something. You should always push the activity of manipulating data to the data layer. In an N-Tier system, the term Tier implies the system has concerns separated into layers. Cluttering up a system with concerns all over the place will make a nightmare Brownfield System. However if you are unable and are forced to compose the object at your layer (which is not unusual), these techniques tend to be extensible. This type of scenario occurs if you are working with disparate systems. For example, half of the data may come from a SQL like data source and the other from a SOAP Service.

Proper Sub collections, Unions, and Intersections are terms used in a branch of mathematics referred to as Set Theory. It describes membership status of items within sets. Sets are identified generically by a letter. The term union is identified by the symbol ∪, Intersection ∩, sub sets and super sets are defined by ⊂ and ⊃ respectfully.

To illustrate a union consider the set A and B. When the two are union-ed they make the following new set called C. This is akin to the mathematical function plus.



Conversely when A intersects with B, C is created. As you can see, C is the difference of the two sets. This is very similar to the minus function.



A Sub Collection and Proper Sub Collection are a little more difficult to understand. While in this example A contains B making it a Proper Sub Collection of A. if B contained everything in A it would not be proper sub collection anymore, It would only be a sub collection.


Example : Unions in Java with CollectionUtils.union


Lets look at Unions and how this can be done in Java with the Apache Commons CollectionUtils.union.

The Code:

package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import java.util.*;
import org.apache.commons.lang.*;
public abstract class AbstractCollectionExampleUtils
{
protected static List<Integer> A = Arrays.asList(1, 2, 3, 4);
protected static List<Integer> B = Arrays.asList(3, 4);
protected static List<Integer> C = Arrays.asList(5, 6);
protected static List<Integer> D = Arrays.asList(1, 2, 3, 4, 5);
protected static List<Integer> E = Arrays.asList();
/**
* Intersection = cap
*/
protected static String intersection() { return "∩"; }
/**
* Union = cup
*/
protected static String union() { return "∪"; }
/**
* Subset of
*/
protected static String subsetOf(){ return "⊂"; }
/**
* Superset of
*/
protected static String supersetOf(){ return "⊃"; }
/**
* Not a subset of
*/
protected static String notASubsetOf(){ return "⊄"; }
/**
* Subset of or equal to
*/
protected static String subSetOfOrEqualTo(){ return "⊆"; }
/**
* Superset of or equal to
*/
protected static String superSetOfOrEqualTo(){ return "⊇"; }
/**
* Print Set
*/
protected static String set(Collection<Integer> S)
{
return " {" + StringUtils.join(S.iterator(), ",") + "} ";
}
}
package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import static org.apache.commons.lang.StringUtils.rightPad;
import java.util.*;
import org.apache.commons.collections.*;
public class Union extends AbstractCollectionExampleUtils
{
public static void main(String[] args)
{
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(B), 13) + "=" + union(A, B));
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(C), 13) + "=" + union(A, C));
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(D), 13) + "=" + union(A, D));
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(E), 13) + "=" + union(A, E));
}
private static String union(Collection s, Collection ss)
{
return set(CollectionUtils.union(s, ss));
}
}


The Results:

When {1,2,3,4} ∪ {3,4} = {1,2,3,4}
When {1,2,3,4} ∪ {5,6} = {1,2,3,4,5,6}
When {1,2,3,4} ∪ {1,2,3,4,5} = {1,2,3,4,5}
When {1,2,3,4} ∪ {} = {1,2,3,4}


Example : Intersection in Java with CollectionUtils.intersection


Now, lets look at intersections and how this can be done in Java with the Apache Commons CollectionUtils.intersection.

The Code:

package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import static org.apache.commons.lang.StringUtils.rightPad;
import java.util.*;
import org.apache.commons.collections.*;
public class Intersection extends AbstractCollectionExampleUtils
{
public static void main(String[] args)
{
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(B),13) + " " + intersection(A, B));
System.out.println("When" + rightPad(set(B),13) + intersection() + rightPad(set(A),13) + " " + intersection(B, A));
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(C),13) + " " + intersection(A, C));
System.out.println("When" + rightPad(set(C),13) + intersection() + rightPad(set(A),13) + " " + intersection(C, A));
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(D),13) + " " + intersection(A, D));
System.out.println("When" + rightPad(set(D),13) + intersection() + rightPad(set(A),13) + " " + intersection(D, A));
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(E),13) + " " + intersection(A, E));
System.out.println("When" + rightPad(set(E),13) + intersection() + rightPad(set(A),13) + " " + intersection(E, A));
}
private static String intersection(Collection s, Collection ss)
{
return "CollectionUtils.intersection(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ") = " + rightPad(set(CollectionUtils.intersection(s, ss)),11);
}
}


The Results:

When {1,2,3,4} ∩ {3,4} CollectionUtils.intersection( {1,2,3,4} , {3,4} ) = {3,4}
When {3,4} ∩ {1,2,3,4} CollectionUtils.intersection( {3,4} , {1,2,3,4} ) = {3,4}
When {1,2,3,4} ∩ {5,6} CollectionUtils.intersection( {1,2,3,4} , {5,6} ) = {}
When {5,6} ∩ {1,2,3,4} CollectionUtils.intersection( {5,6} , {1,2,3,4} ) = {}
When {1,2,3,4} ∩ {1,2,3,4,5} CollectionUtils.intersection( {1,2,3,4} , {1,2,3,4,5} ) = {1,2,3,4}
When {1,2,3,4,5} ∩ {1,2,3,4} CollectionUtils.intersection( {1,2,3,4,5} , {1,2,3,4} ) = {1,2,3,4}
When {1,2,3,4} ∩ {} CollectionUtils.intersection( {1,2,3,4} , {} ) = {}
When {} ∩ {1,2,3,4} CollectionUtils.intersection( {} , {1,2,3,4} ) = {}


Example : Sub collections and Proper Sub collections in Java with CollectionUtils.containsAny, isProperSubCollection, and isSubCollection


The first two examples were simple enough. Let’s look at Sub collection versus Proper Sub Collection. We will use the containsAny, isProperSubCollection and isSubCollection static methods off the CollectionUtils class.


package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import java.util.*;
import static org.apache.commons.lang.StringUtils.rightPad;
import org.apache.commons.collections.*;
public class SubCollections extends AbstractCollectionExampleUtils
{
public static void main(String[] args)
{
System.out.println("Intersection Tests : ");
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(B), 13) + " " + containsAny(A, B));
System.out.println("Is" + rightPad(set(B), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(B, A));
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(C), 13) + " " + containsAny(A, C));
System.out.println("Is" + rightPad(set(C), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(C, A));
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(D), 13) + " " + containsAny(A, D));
System.out.println("Is" + rightPad(set(D), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(D, A));
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(E), 13) + " " + containsAny(A, E));
System.out.println("Is" + rightPad(set(E), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(E, A));
System.out.println("Is" + rightPad(set(E), 13) + intersection() + rightPad(set(E), 13) + " " + containsAny(E, E));
System.out.println("");
System.out.println("Subsets Tests: " );
System.out.println("B " + subsetOf() + " A indicates B is a subset of A, but are not equal. When B is equal to A it is usually denoted as B " + subSetOfOrEqualTo() + " A");
System.out.println("");
System.out.println("Is" + rightPad(set(A), 13) + subsetOf() + rightPad(set(A), 13) + " " + subsetOfOrEqualTo(A, A));
System.out.println("Is" + rightPad(set(B), 13) + subsetOf() + rightPad(set(A), 13) + " " + subsetOfOrEqualTo(B, A));
System.out.println("Is" + rightPad(set(C), 13) + subsetOf() + rightPad(set(A), 13) + " " + subsetOfOrEqualTo(C, A));
System.out.println("");
System.out.println("Is" + rightPad(set(A), 13) + subSetOfOrEqualTo() + rightPad(set(A), 13) + " " + superSetOfOrEqualTo(A, A));
System.out.println("Is" + rightPad(set(B), 13) + subSetOfOrEqualTo() + rightPad(set(A), 13) + " " + superSetOfOrEqualTo(B, A));
System.out.println("Is" + rightPad(set(C), 13) + subSetOfOrEqualTo() + rightPad(set(A), 13) + " " + superSetOfOrEqualTo(C, A));
}
protected static String containsAny(Collection s, Collection ss)
{
return rightPad(" CollectionUtils.containsAny(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ")", 67) + " = " + String.valueOf(CollectionUtils.containsAny(s, ss));
}
protected static String subsetOfOrEqualTo(Collection s, Collection ss)
{
return rightPad(" CollectionUtils.isProperSubCollection(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ")", 67) + " = " + String.valueOf(CollectionUtils.isProperSubCollection(s, ss));
}
private static String superSetOfOrEqualTo(Collection s, Collection ss)
{
return rightPad(" CollectionUtils.isSubCollection(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ")", 67) + " = " + String.valueOf(CollectionUtils.isSubCollection(s,ss));
}
}


The Results:

Intersection Tests :
Is {1,2,3,4} ∩ {3,4} CollectionUtils.containsAny( {1,2,3,4} , {3,4} ) = true
Is {3,4} ∩ {1,2,3,4} CollectionUtils.containsAny( {3,4} , {1,2,3,4} ) = true
Is {1,2,3,4} ∩ {5,6} CollectionUtils.containsAny( {1,2,3,4} , {5,6} ) = false
Is {5,6} ∩ {1,2,3,4} CollectionUtils.containsAny( {5,6} , {1,2,3,4} ) = false
Is {1,2,3,4} ∩ {1,2,3,4,5} CollectionUtils.containsAny( {1,2,3,4} , {1,2,3,4,5} ) = true
Is {1,2,3,4,5} ∩ {1,2,3,4} CollectionUtils.containsAny( {1,2,3,4,5} , {1,2,3,4} ) = true
Is {1,2,3,4} ∩ {} CollectionUtils.containsAny( {1,2,3,4} , {} ) = false
Is {} ∩ {1,2,3,4} CollectionUtils.containsAny( {} , {1,2,3,4} ) = false
Is {} ∩ {} CollectionUtils.containsAny( {} , {} ) = false

Subsets Tests:
B ⊂ A indicates B is a subset of A, but are not equal. When B is equal to A it is usually denoted as B ⊆ A

Is {1,2,3,4} ⊂ {1,2,3,4} CollectionUtils.isProperSubCollection( {1,2,3,4} , {1,2,3,4} ) = false
Is {3,4} ⊂ {1,2,3,4} CollectionUtils.isProperSubCollection( {3,4} , {1,2,3,4} ) = true
Is {5,6} ⊂ {1,2,3,4} CollectionUtils.isProperSubCollection( {5,6} , {1,2,3,4} ) = false

Is {1,2,3,4} ⊆ {1,2,3,4} CollectionUtils.isSubCollection( {1,2,3,4} , {1,2,3,4} ) = true
Is {3,4} ⊆ {1,2,3,4} CollectionUtils.isSubCollection( {3,4} , {1,2,3,4} ) = true
Is {5,6} ⊆ {1,2,3,4} CollectionUtils.isSubCollection( {5,6} , {1,2,3,4} ) = false


The Apache Commons Collections has some enormous potential. I find I uses these static methods frequently. I hope this has helped you in solving your problems, please feel free to drop me a line if you have any questions.

Author: Philip A Senger

Saturday, February 7, 2009

Examples of Functors, Transformers, Predicates, and Closures in Java

One day, I found myself re-designing a procurement portal, and I kept re-writing the same for-loop over and over again (no pun intended). I had an epiphany; I could do better and I started using the Apache Commons Collection Utilities (Transformers, Predicates, and Closures). Now don’t think just because I started to use the Apache Commons Collection Utilities, the project was better. However, the result was a highly extensible framework…. Later it was dismantled by another team… but that is a different story (Grin).

Functors

I laugh every time I think of the word Functors, but that is because I’m immature, case in point, I still laugh at fart jokes. Anyway, Functors, or Function Objects, in the Apache or Jakarta Commons Collection Utilities are a set of interfaces designed specifically to be used against collections of objects. This framework embodies a balance between code reuse and behavioral specialization through composition as opposed to strict Object Oriented design. Composition is well suited for Creational patterns such as Factories, Structural patterns like Decorators, or Behavioral patterns like Strategies. The Apache Commons Collections framework defines three types of interfaces:
  • Closures are functions that can alter the object and get a reference to each object in the collection.

  • Transformers are responsible for transforming data from one format to another or from one object to another.

  • Predicates simply execute a conditional test against each item in a collection and return true or false for each item.
NOTE:
My examples sometimes use Anonymous Inner Classes and Inner Classes. Some developers have strong feelings about defining classes in this manner. There are times when doing this is appropriate and times when it is inappropriate. As with any programming solutions, this technique may or may not suit your needs or environment. So, let’s get over it and move on.

Closure

Here is a typical problem statement which maybe resolved with the use of Closures. I want to execute a specific method or change the state on every object in a collection. For example, I might want to execute the toString method. Please note that these examples do not pull out a value and transform into another collection of objects. They simply iterate over the collection and do something to it or with it. For the purpose of this first example, I will send the results of the toString to system.out. In the second example, I will alter the state of each bean and change the name of every object. Both examples use the utility method CollectionUtils.forAllDo.

Lets take some baby steps, and look at this example.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
import org.apache.commons.lang.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class SimpleClosure
{
public static void main(String[] args)
{
System.out.println("\nTest Number One Results :");
List<String> collectionOfWords = Arrays.asList("Java", "Example",
"Help", "Tips", "And",
"Tricks", "Apache",
"Commons", "Collections");
// Lets call toString on every object and print it out.
CollectionUtils.forAllDo(collectionOfWords, new Closure()
{
public void execute(Object o)
{
assert o != null;
System.out.print(o.toString() + " ");
}
});
System.out.println("\n\nTest Number Two Results :");
int i = 1;
List<DTO> collectionOfDTOs = Arrays.asList(new DTO(i++, "Java Tips and Tricks", Gender.Male, State.WI),
new DTO(i++, "Apache Commons" , Gender.Male, State.WI),
new DTO(i++, "Jakarta Commons" , Gender.Male, State.WI),
new DTO(i++, "Collections" , Gender.Male, State.WI),
new DTO(i++, "Closures" , Gender.Male, State.WI) );
CollectionUtils.forAllDo(collectionOfDTOs, new Closure()
{
public void execute(Object o)
{
DTO dto = (DTO) o;
assert dto != null;
String s = StringUtils.defaultIfEmpty(dto.getName(), "null");
dto.setName("Yoda says, " + s + " Rocks!");
}
});
CollectionUtils.forAllDo(collectionOfDTOs,PrintIt.getInstance());
}
}
The Results:
Test Number One Results :
Java Example Help Tips And Tricks Apache Commons Collections

Test Number Two Results :
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Yoda says, Java Tips and Tricks Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Yoda says, Apache Commons Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Yoda says, Jakarta Commons Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Yoda says, Collections Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Yoda says, Closures Rocks!', gender=Male, state=WI}

You might have noticed I created a class called PrintIt and it has a static method called getInstance. This class is an implementation of the singleton design pattern. In a large system where Closures (Predicates and Transformers) are being created by the hundreds, it makes sense to keep the memory foot print to a minimum. This is accomplished by using a singleton pattern. Word of caution, there is only one instance of this class per Java Virtual Machine (jvm), so using static properties could cause big problems, if you don’t understand how they work.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import org.apache.commons.collections.*;
public class PrintIt implements Closure
{
// This class implements a Singleton Pattern
private static PrintIt ourInstance = new PrintIt();
/**
* Get a singleton instance of PrintIt
*/
public static PrintIt getInstance()
{
return ourInstance;
}
private PrintIt() // This is a singleton, dont change this!
{
}
public void execute(Object o)
{
System.out.println( o.toString() );
}
}
Once again, in the first example, all we did was iterate through a collection and call toString on every element in the collection. In the second example, we are actually modifying the state of the bean by rewriting the name of the DTO.. I love Yoda! Lets talk about transformers now.

Transformer

I’m a little older than the Transformer cartoons, but I cant help but think of the movie that was released not long ago. So, here is a problem statement that would best be resolved by transformers. You just have a whole bunch of string values from a Http Request object and you need to convert them to Integers.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class SimpleTransformer
{
public static void main(String[] args)
{
Collection<String> stringOfNumbers = Arrays.asList("1", "2", "3", "4");
Collection<Integer> intNums = CollectionUtils.collect(stringOfNumbers, new Transformer() {
public Object transform(Object o) {
return Integer.valueOf((String) o);
}
});
CollectionUtils.forAllDo(intNums, PrintIt.getInstance() );
}
}
The Results:
1
2
3
4
Again, I used the PrintIt class to print the results, but you get the idea with this example. I converted a collection of Strings to Integers. Of course there are many ways to skin a cat, this is just an example. You might have noticed by now, that the interface is public Object transform(Object o)… It is not a Java 1.5 Generics implementation. I’m not entirely sure why the Apache team hasn’t released a Java 1.5 Generics version of these utilities, but if you are in a pinch and you have to have it, someone posted a link to an adaptation of the library that has the Generics. http://larvalabs.com/collections

Here is a more practical problem statement. You have a collection of plain old java beans and in each bean a method that returns a String, part of the String represents the id into another system, environment or sub identity. For example, the String might be prefixed with three letters, for example "PAS". You might find some legacy ERP systems that do this to the PO number. Maybe the billing department puts the initials of the sales person or team at the front of the number. In this example we will transform an Array of Strings that have ids in them into an array of ids in the form of numbers only.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
import org.apache.commons.lang.*;
public class SimpleTransformer
{
public static void main(String[] args)
{
Collection<String> stringOfNumbers = Arrays.asList("ABC0001", "BCD0002", "CDF0003", "BFA0004");
Collection<Integer> intNums = CollectionUtils.collect(stringOfNumbers, new Transformer()
{
public Object transform(Object o)
{
String s = ((String) o);
return Integer.valueOf(s.substring(3, s.length()));
}
});
CollectionUtils.forAllDo(intNums, PrintIt.getInstance());
}
}
The Results:
1
2
3
4
Now let’s look at something a little more practical. As a problem statement, let’s say we have a billing object from the old system called OldBill, and we need to identify it in the new system, called NewBill. The ids in the old system started with “A” and a number. In the new system they will start with “Z” and the number from the old system plus 500. In the next example we will break apart the concerns of the transformers into two different transformers and glue them together with the utility class TransformerUtils.chainedTransformer. Creating a transformer in this manner allows us to plug in a different behavior or add and subtract behaviors, even on the fly.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class ChainedTransformer
{
public static void main(String[] args)
{
List<OldBill> aList = Arrays.asList(new OldBill("A1"), new OldBill("A2"),
new OldBill("A3"), new OldBill("A4"));
Transformer[] chainedTransformer = new Transformer[]{
new Transformer() {
public Object transform(Object o) {
return ((OldBill )o).getId().replace('A', 'Z');
}
},
new Transformer() {
public Object transform(Object o) {
char[] c = ((String) o).toCharArray();
int x = Integer.parseInt(String.valueOf(c[1])) + 500;
return new NewBill( String.valueOf(c[0]) + x );
}
}
};
System.out.println("The aList");
CollectionUtils.forAllDo(aList, PrintIt.getInstance());
List<NewBill> bList = (List<NewBill>) CollectionUtils.collect(aList, TransformerUtils.chainedTransformer(chainedTransformer));
System.out.println("\nThe bList");
CollectionUtils.forAllDo(bList, PrintIt.getInstance());
}
}
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
public class OldBill
{
private String id;
public OldBill(String id)
{
this.id = id;
}
public String getId()
{
return id;
}
public void setId(String id)
{
this.id = id;
}
@Override public String toString()
{
return "OldBill{id='" + id + "\'}";
}
}
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
public class NewBill
{
private String id;
public NewBill(String id)
{
this.id = id;
}
public String getId()
{
return id;
}
public void setId(String id)
{
this.id = id;
}
@Override public String toString()
{
return "NewBill{id='" + id + "\'}";
}
}
The Results:
The aList
OldBill{id='A1'}
OldBill{id='A2'}
OldBill{id='A3'}
OldBill{id='A4'}

The bList
NewBill{id='Z501'}
NewBill{id='Z502'}
NewBill{id='Z503'}
NewBill{id='Z504'}
Predicate

Predicates do one thing and one thing only, they return either true or false. As a problem statement, let’s say we have a collection of Strings and we want keep out values that can not be converted to numbers.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class SimplePredicate
{
public static void main(String[] args)
{
List<String> mixedup = Arrays.asList("A", "0", "B", "C", "1", "D", "F", "3");
Collection numbersOnlyList = CollectionUtils.predicatedCollection(new ArrayList(),
new Predicate() {
public boolean evaluate(Object o) {
try {
Integer.valueOf((String) o);
return true;
} catch (NumberFormatException e) {
return false;
}
}
});
for (String s : mixedup) {
try {
numbersOnlyList.add(s);
} catch (IllegalArgumentException e) {
System.out.println("I love CollectionUtils!");
}
}
System.out.println("\nResults of the predicatedCollection List:");
CollectionUtils.forAllDo(numbersOnlyList, PrintIt.getInstance() );
}
}
The Results:
I love CollectionUtils!
I love CollectionUtils!
I love CollectionUtils!
I love CollectionUtils!
I love CollectionUtils!

Results of the predicatedCollection List:
0
1
3
Ok here is something more useful. Have you ever had a need to do something like SQL but in Java? For example, you wanted to select beans from a collection based on some conditional blocks? Here are some great examples of SQL commands that can be used in Java and they behave just like SQL complex where clauses, distinct, like, and group by. In this example, we will be using the folloing utlitly classes and maps
The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
import org.apache.commons.collections.map.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class PredicatesSQLSample
{
public static void main(String[] args)
{
List<DTO> list = Arrays.asList(new DTO(1,"Bob", Gender.Male, State.WI), new DTO(2,"Larry",Gender.Male, State.WI),
new DTO(3,"Bill", Gender.Male, State.WI), new DTO(4,"Sue", Gender.Female, State.AZ),
new DTO(3,"Bill", Gender.Male, State.WI), new DTO(4,"Sue", Gender.Female, State.AZ),
new DTO(5,"Joe", Gender.Male, State.AZ), new DTO(6,"Zoe", Gender.Female, State.MI));
Predicate sqlOrQueryPredicate = PredicateUtils.anyPredicate(new Predicate[]{
new Predicate()
{
public boolean evaluate(Object o)
{
return State.WI.equals(((DTO) o).getState());
}
}, new Predicate()
{
public boolean evaluate(Object o)
{
return Gender.Female.equals(((DTO) o).getGender());
}
}
});
Predicate sqlAndQueryPredicate = PredicateUtils.allPredicate(new Predicate[]{
new Predicate()
{
public boolean evaluate(Object o)
{
return State.AZ.equals(((DTO) o).getState());
}
}, new Predicate()
{
public boolean evaluate(Object o)
{
return Gender.Male.equals(((DTO) o).getGender());
}
}
});
Predicate likeNameStartsWithB = new Predicate(){
public boolean evaluate(Object o)
{
return ((DTO) o).getName().startsWith("B");
}
};

Collection aList = CollectionUtils.select(list, sqlOrQueryPredicate);
Collection bList = CollectionUtils.select(list, PredicateUtils.notPredicate( sqlOrQueryPredicate ));
Collection cList = CollectionUtils.select(list, sqlAndQueryPredicate);
Collection dList = CollectionUtils.select(list, PredicateUtils.allPredicate(new Predicate[]{PredicateUtils.uniquePredicate(), sqlOrQueryPredicate} ));
Collection eList = CollectionUtils.select(list, PredicateUtils.allPredicate(new Predicate[]{PredicateUtils.uniquePredicate() ,likeNameStartsWithB} ));
Collection fList = CollectionUtils.select(list, PredicateUtils.uniquePredicate() );

Map aGroupByStateMap = TransformedMap.decorate(new MultiValueMap(),new Transformer(){
public Object transform(Object o)
{
return ((DTO) o).getState();
}
}, TransformerUtils.nopTransformer() );
for (Object o : fList)
{
aGroupByStateMap.put( o, o );
}

System.out.println("\nAll the people :\nselect * from list");
CollectionUtils.forAllDo(list,PrintIt.getInstance());
System.out.println("\nAll the people in Wisconsin OR Female :\nselect * from list where ( state = WI or gender = female );");
CollectionUtils.forAllDo(aList, PrintIt.getInstance());
System.out.println("\nAll the people NOT ( Wisconsin OR Female ) :\nselect * from list where ! ( state = WI or gender = female );");
CollectionUtils.forAllDo(bList, PrintIt.getInstance());
System.out.println("\nAll the people in Arizona AND Male :\nselect * from list where ( state = AZ and gender = male );");
CollectionUtils.forAllDo(cList, PrintIt.getInstance());
System.out.println("\nAll the distinct people in Arizona AND Male :\nselect distinct * from list where ( state = WI or gender = female );");
CollectionUtils.forAllDo(dList, PrintIt.getInstance());
System.out.println("\nAll the distinc people with the name that starts with B :\nselect distinct * from list where name like \"B%\";");
CollectionUtils.forAllDo(eList, PrintIt.getInstance());
System.out.println("\nAll the distinct people grouped by state :\nselect distinct * from list group by state;");
Set states = aGroupByStateMap.keySet();
for (Object state : states)
{
System.out.println(((State)state).getFullyQualifiedName());
CollectionUtils.forAllDo((Collection) aGroupByStateMap.get(state), PrintIt.getInstance());
}
}
}
The Results:
All the people :
select * from list
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}

All the people in Wisconsin OR Female :
select * from list where ( state = WI or gender = female );
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}

All the people NOT ( Wisconsin OR Female ) :
select * from list where ! ( state = WI or gender = female );
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}

All the people in Arizona AND Male :
select * from list where ( state = AZ and gender = male );
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}

All the distinct people in Arizona AND Male :
select distinct * from list where ( state = WI or gender = female );
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}

All the distinc people with the name that starts with B :
select distinct * from list where name like "B%";
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}

All the distinct people grouped by state :
select distinct * from list group by state;
ARIZONA
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}
MICHIGAN
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}
WISCONSIN
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}

Well, that wraps it up. Next week I plan on showing you some more examples of this fantastic api called Apache Commons.

Author: Philip A Senger