.. _whatsnew-0.3:

==========================
What's New in Astropy 0.3?
==========================

Overview
--------

Astropy 0.3 is a major release that adds significant new functionality since
the 0.2.x series of releases. This release sees the addition of the `Modeling`_
sub-package, which provides an initial framework for fitting models to data,
and the `Virtual Observatory`_ sub-package, which implements cone searches. In
addition, there have been many improvements to the existing
sub-packages, such as a much faster implementation of quantities for Numpy
arrays in the `Units and Quantities`_ sub-package, a better integration of
units and quantities in other sub-packages (such as the `Cosmology`_
sub-package), new `Table`_ functionality related to joining and aggregation,
support for arrays in the `Coordinates`_ sub-package, and a re-structuring of
the `Convolution`_ code to provide many build-in parameterized kernels. We
describe the new sub-packages and the main improvements below.

For a detailed list of all the changes, including deprecated features and backward-incompatible changes,
please take a look at the full :doc:`../changelog`.

Modeling
--------

The :ref:`astropy-modeling` sub-package defines many built-in 1-D and 2-D models and
provides an extensible framework for fitting these to data. The following
example demonstrates how to easily define a Gaussian model and fit it to a
previously defined dataset:

.. plot::
   :include-source:

    import numpy as np
    from astropy.modeling import models, fitting

    # Generate fake data
    np.random.seed(0)
    x = np.linspace(-5., 5., 200)
    y = 3 * np.exp(-0.5 * (x - 1.3)**2 / 0.8**2)
    y += np.random.normal(0., 0.2, x.shape)

    # Fit the data
    g_init = models.Gaussian1D(amplitude=1., mean=0, stddev=1.)
    f2 = fitting.NonLinearLSQFitter()
    g = f2(g_init, x, y)

    # Plot the results
    plt.figure(figsize=(8,5))
    plt.plot(x, y, 'ko')
    plt.plot(x, g(x), 'r-', lw=2)
    plt.xlabel('Position')
    plt.ylabel('Flux')

.. note:: Since this is a new sub-package, it should be considered experimental
          and will be significantly improved in future. In particular, the
          ability to fit composite models (such as the sum of two models) will
          be added in the next major version (in the mean time, users can
          define their own models for such cases). More complete examples and a
          full description of the sub-package is provided in
          :ref:`astropy-modeling`.

Virtual Observatory
-------------------

The new :ref:`astropy_vo` sub-package currently provides the ability to execute
cone searches. To see a list of available catalogs, you can first import the
cone search module::

    >>> from astropy.vo.client import conesearch

then make use of the :func:`~astropy.vo.client.conesearch.list_catalogs` function::

    >>> conesearch.list_catalogs()
    [u'Guide Star Catalog 2.3 1',
     u'SDSS DR7 - Sloan Digital Sky Survey Data Release 7 1',
     u'SDSS DR7 - Sloan Digital Sky Survey Data Release 7 2',
     u'SDSS DR7 - Sloan Digital Sky Survey Data Release 7 3',
     u'SDSS DR7 - Sloan Digital Sky Survey Data Release 7 4',
     u'SDSS DR8 - Sloan Digital Sky Survey Data Release 8 1',
     u'SDSS DR8 - Sloan Digital Sky Survey Data Release 8 2',
     u'The HST Guide Star Catalog, Version 1.1 (Lasker+ 1992) 1',
     u'The HST Guide Star Catalog, Version 1.2 (Lasker+ 1996) 1',
     u'The HST Guide Star Catalog, Version GSC-ACT (Lasker+ 1996-99) 1',
     u'The PMM USNO-A1.0 Catalogue (Monet 1997) 1',
     u'The USNO-A2.0 Catalogue (Monet+ 1998) 1',
     u'Two Micron All Sky Survey (2MASS) 1',
     u'Two Micron All Sky Survey (2MASS) 2',
     u'USNO-A2 Catalogue 1',
     u'USNO-A2.0 1']


The following example shows how to execute a cone search for a radius of 0.1
degrees around M31 from the 2MASS catalog. First, we can extract the
coordinates for M31 from SIMBAD::

    >>> from astropy import coordinates as coords
    >>> c = coords.ICRS.from_name('M31')
    >>> c
    <ICRS RA=10.68471 deg, Dec=41.26875 deg>

and we can then execute the query::

    >>> from astropy import units as u
    >>> twomass = 'Two Micron All Sky Survey (2MASS) 1'
    >>> result = conesearch.conesearch(c, 0.1 * u.degree, catalog_db=twomass)
    Trying http://wfaudata.roe.ac.uk/twomass-dsa/DirectCone?DSACAT=TWOMASS&...
    Downloading ...

.. in the following paragraph, we deliberately omit the ~ because the class
.. names are the same otherwise.

The result is returned as a :class:`astropy.io.votable.tree.Table` instance
(note that this is different from the generic
:class:`astropy.table.table.Table` class)::

    >>> result.url
    u'http://wfaudata.roe.ac.uk/twomass-dsa/DirectCone?DSACAT=TWOMASS&DSATAB=twomass_psc&'
    >>> result.array.size
    2008
    >>> result.array['ra']
    masked_array(data = [10.620983 10.672264 10.651166 ..., 10.805599],
                 mask = [False False False ..., False],
           fill_value = 1e+20)

.. as above, we are deliberately not using ~ in the API link

Converting to an :class:`astropy.table.table.Table` is
straightforward::

    >>> table = result.to_table()
    >>> print(table)
          cx             cy             cz       ... coadd_key coadd
    -------------- -------------- -------------- ... --------- -----
    0.739345466303 0.138832922929 0.658857876152 ...   1590591    33
     0.73937055914 0.138481636014 0.658903644522 ...    577809    33
    0.739301283105 0.138762751963 0.658922234764 ...   1590591    33
               ...            ...            ... ...       ...   ...
    0.739389950758 0.139376220174 0.658693229028 ...   1590591    33
    0.739099087292 0.140726588258 0.658732545516 ...   1590631   232
    0.737999550267 0.140950272324 0.659916422388 ...   1590632   244

Units and Quantities
--------------------

Quantity instances in the :ref:`astropy-units` sub-package are now
fully-fledged Numpy arrays, and common Numpy functions (such as
`numpy.mean`, `numpy.cos`, `numpy.log10`, etc.) will now
correctly treat the units:

    >>> q = np.array([1., 2., 3., 4.]) * u.m / u.s
    >>> np.mean(q)
    <Quantity 2.5 m / s>
    >>> np.std(q)
    <Quantity 1.11803398875 m / s>

This includes functions that only accept specific units such as angles::

    >>> q = 30. * u.deg
    >>> np.sin(q)
    <Quantity 0.5 >

or dimensionless quantities::

    >>> nu = 3 * u.GHz
    >>> T = 30 * u.K
    >>> np.exp(- h * nu / (k_B * T))
    <Quantity 0.995212254619 >

.. note:: Not *all* Numpy functions (in particular non-ufuncs) and functions
          outside Numpy will treat units correctly, so be aware that units may
          be implicitly dropped without a warning. Always check that the
          function you are using treats the units correctly.

Another change is that imperial units are not enabled by default. To enable
them, use::

    >>> from astropy.units import imperial
    >>> imperial.enable()

Coordinates
-----------

Coordinate objects from the :ref:`astropy-coordinates` sub-package can now
store arrays of coordinates instead of just a single coordinate. This
dramatically speeds up coordinate conversions when many coordinates are used.
The following example shows how one can combine this with functionality from
the :class:`~astropy.table.table.Table` class to read in arrays of coordinates
and convert them to a different coordinate frame. First, we can read in a table
which contains coordinates both in decimal and string form::

    >>> from astropy.table import Table
    >>> t = Table.read('2mass.tbl', format='ascii.ipac')
    >>> print(t)
        ra        dec         sra           sdec     ...  h_k   j_k
    ---------- ---------- ------------ ------------- ... ----- -----
    274.429506 -13.870547 18h17m43.08s -13d52m13.97s ... 0.791 3.048
    274.423821  -13.86974 18h17m41.72s -13d52m11.06s ... 0.867 3.034
    274.424587 -13.739629 18h17m41.90s -13d44m22.66s ...  0.94    --
           ...        ...          ...           ... ...   ...   ...
    274.870009 -13.817775 18h19m28.80s -13d49m03.99s ... 1.557    --
    274.735323 -13.941575 18h18m56.48s -13d56m29.67s ...    --    --
    274.866294 -13.841778 18h19m27.91s -13d50m30.40s ... 1.146    --

Then we can create the coordinate object either with the decimal floating-point
values::

    >>> from astropy import units as u
    >>> from astropy import coordinates as coords
    >>> c = coords.FK5(t['ra'], t['dec'], unit=(u.deg, u.deg))
    >>> c[0]
    <FK5 RA=274.42951 deg, Dec=-13.87055 deg>
    >>> c[1]
    <FK5 RA=274.42382 deg, Dec=-13.86974 deg>

or the string values::

    >>> c = coords.FK5(t['sra'], t['sdec'], unit=(u.deg, u.deg))
    >>> c[0]
    <FK5 RA=274.42950 deg, Dec=-13.87055 deg>
    >>> c[1]
    <FK5 RA=274.42383 deg, Dec=-13.86974 deg>

Note that in the second case, the initialization will be slower due to the
parsing of the strings. These coordinates can then easily be converted to other frames::

    >>> cgal = c.transform_to(coords.Galactic)
    >>> cgal[0]
    <Galactic l=16.77345 deg, b=0.99446 deg>
    >>> cgal[1]
    <Galactic l=16.77155 deg, b=0.99968 deg>

For coordinate arrays, accessing attributes such as ``ra`` or ``dec`` will now
return Numpy arrays.

As shown above, all the coordinate classes have now been renamed to drop the
``Coordinates`` suffix (e.g. ``ICRS`` instead of ``ICRSCoordinates``). In
addition, `HorizontalCoordinates` has now been renamed to `AltAz`.

Coordinate objects now support matching one set of coordinates to another
using the array coordinates functionality described above.  See
`~astropy.coordinates.coordsystems.SphericalCoordinatesBase.match_to_catalog_sky`
and `~astropy.coordinates.coordsystems.SphericalCoordinatesBase.match_to_catalog_3d`
for more details.

Finally, coordinate objects now have a ``to_string`` method that allows easy
conversion to string representations. For example, in the case of the previous
coordinates used above::

    >>> c.to_string()
    [u'18h17m43.08s -13d52m13.97s',
     u'18h17m41.72s -13d52m11.06s',
     ...
     u'18h18m56.48s -13d56m29.67s',
     u'18h19m27.91s -13d50m30.4s']

Table
-----

In addition to many bug fixes and usability improvements, the key new feature
in the :ref:`astropy-table` sub-package is the addition of high level
:ref:`table_operations` that can be used to generate a new table from one or
more input tables:

.. list-table::
   :header-rows: 1
   :widths: 28 52 20

   * - Documentation
     - Description
     - Function
   * - :ref:`grouped-operations`
     - Group tables and columns by keys
     - `~astropy.table.table.Table.group_by`
   * - :ref:`stack-vertically`
     - Concatenate input tables along rows
     - `~astropy.table.operations.vstack`
   * - :ref:`stack-horizontally`
     - Concatenate input tables along columns
     - `~astropy.table.operations.hstack`
   * - :ref:`table-join`
     - Database-style join of two tables
     - `~astropy.table.operations.join`

Grouping is a useful concept that allows you to divide a table
into sub-groups based on certain key values and create new tables based
on computed properties of those sub-groups.  As an example, if you
have a table containing photometric observations of multiple sources
over multiple epochs, it would be possible to compute a mean magnitude
for each unique object.  In addition to this *aggregation* operation,
the grouping interface also allows *filtering* operations where certain
groups are excluded from the resultant table.

Even more powerful is the ability to do database-style joins of tables.  For
instance, if you have distinct tables with photometry in different wavebands
for a set of objects, you join these points into a single table with one row
for each source.

Time
----

The :ref:`astropy-time` sub-package has received attention in filling in
the details from the initial release in astropy 0.2.  This includes adding array
indexing and supporting various arithmetic operations involving arrays,
constants, and `~astropy.units.quantity.Quantity` objects with time units.  In addition the initial
infrastructure was added to allow use of `International Earth Rotation and
Reference Systems Service
<http://www.iers.org/>`_ tables so that automatic
calculation of UT1 becomes possible.

One very significant improvement is an overhaul of the internal
time manipulations so that arithmetic with `~astropy.time.core.Time` and
`~astropy.time.core.TimeDelta` objects maintain sub-nanosecond precision over a time span
longer than the age of the universe.  This is done by carefully managing
the way the time is represented and manipulated using two 64-bit floats.

Finally, three new time formats were added:

  - ``datetime``: standard library `datetime.datetime` objects.
  - ``plot_date``: dates compatible with the `matplotlib.pyplot.plot_date` function.
  - ``gps``: seconds since 1980-01-01 00:00:00 UTC including leap seconds.

ASCII Tables
------------

The :ref:`io-ascii` sub-package now includes functionality to write `IPAC format tables
<http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html>`_.

The `~astropy.io.ascii.ui.read()` and `~astropy.io.ascii.ui.write()` functions now
allow a ``format`` keyword argument for specifying the file format as a string.
This replaces the deprecated ``Reader`` and ``Writer`` keywords which required
supplying a fully-qualified class type.  To convert existing code that uses
``Reader`` or ``Writer``, change the class name to all lower case with
underscores between words.  For instance::

    >>> from astropy.io import ascii
    >>> data = [' name         age ',
                '-----------   ----',
                'Jane Doe       31 ',
                'John Smith     45 ']
    >>> t = ascii.read(data, Reader=ascii.FixedWidthTwoLine)  # OLD
    >>> t = ascii.read(data, format='fixed_width_two_line')  # NEW

Unified File Read/Write Interface
----------------------------------

FITS format tables can now be read and written via the :ref:`table_io`. 
All of the ASCII table formats are now supported as well.  When using
the unified file interface for ASCII tables the ``format`` defined in the :ref:`io-ascii`
package is prefixed with ``'ascii.'``.  Thus the previous example would be written::

    >>> from astropy.table import Table
    >>> t = Table.read(data, format='ascii.fixed_width_two_line')

Some formats such as ``cds`` or ``latex`` will work without the ``'ascii.'``
prefix but this is deprecated and will be removed in the next major release.

The full list of available formats is now available via the
`~astropy.io.registry.get_formats` function::

    >>> from astropy.io import registry
    >>> print registry.get_formats()
    Data class    Format    Read Write Auto-identify Deprecated
    ---------- ------------ ---- ----- ------------- ----------
         Table        ascii  Yes   Yes            No
         Table ascii.aastex  Yes   Yes            No
         Table  ascii.basic  Yes   Yes            No
         Table    ascii.cds  Yes    No            No
           ...          ...  ...   ...           ...        ...
         Table      daophot  Yes    No            No        Yes
         Table         ipac  Yes   Yes            No        Yes
         Table        latex  Yes   Yes            No        Yes
         Table          rdb  Yes   Yes            No        Yes

If you make a mistake and specify an unavailable or incorrect ``format``, the
error message will now list all the available formats.

Convolution
-----------

The convolution functionality that was originally included in the
:ref:`astropy_nddata` sub-package has now been moved to the new
:ref:`astropy_convolve` sub-package. As part of a Google Summer of Code
`project <http://adonath.github.io/>`_ it has been refactored to include a
framework that provides common built-in kernels:

.. plot::
   :include-source:

    import numpy as np
    from astropy.convolution import convolve, Gaussian2DKernel

    # Generate data
    np.random.seed(0)
    image = np.random.random((128, 128))

    # Create kernel
    g = Gaussian2DKernel(width=1)

    # Convolve data
    image_new = convolve(image, g, boundary='extend')

    # Plot the results
    plt.figure(figsize=(8,3))
    plt.subplot(1,2,1)
    plt.imshow(image, interpolation='none', origin='lower', vmin=0., vmax=1.)
    plt.title('Reference')
    plt.subplot(1,2,2)
    plt.imshow(image_new, interpolation='none', origin='lower', vmin=0., vmax=1.)
    plt.title('Convolved')

A number of different 1-D and 2-D kernels are provided, based on models defined
in :ref:`astropy-modeling`. The discretization of the kernels can be handled in
various ways (e.g. oversampling, interpolation, etc.) which are described in
more detail in :ref:`astropy_convolve`.

Cosmology
---------

The :ref:`astropy-cosmology` sub-package now includes support for including massive
neutrinos in the cosmology classes, and the Planck 2013 cosmology has been
updated to use this. In addition, :class:`~astropy.units.quantity.Quantity`
objects are now used wherever appropriate::

    >>> from astropy.cosmology import WMAP9
    >>> WMAP9.H0
    <Quantity 69.32 km / (Mpc s)>
    >>> WMAP9.lookback_time(3)
    <Quantity 11.590618401420071 Gyr>
    >>> WMAP9.luminosity_distance(3)
    <Quantity 26015.607762091513 Mpc>

Statistics
----------

The :ref:`tools` sub-package includes a number of new common statistical
functions, for example related to binomial statistics and bootstrapping.

WCS
---

When reading FITS headers with the :ref:`astropy-wcs` sub-package, warnings
will now be displayed about any non-standard WCS keywords that were fixed to
become standard compliant.

For users who have `Scipy <http://www.scipy.org>`_ installed, the
:class:`~astropy.wcs.wcs.WCS` class features a new method
:meth:`~astropy.wcs.wcs.WCS.all_world2pix` for converting from world
coordinates to pixel space including the inversion of astrometric distortion
corrections.

The included version of `wcslib` has been upgraded to version 4.19. The
relevant changes for astropy users are:

  * Implemented the butterfly projection (``XPH``), being the polar
    form of the HEALPix projection with ``(H,K) = (4,3)``.

  * Bug fix in ``celfix()`` when translating GLS to SFL with non-zero
    reference point.

  * A number of memory handling and stability fixes.

VO Tables
---------

The :ref:`astropy-io-votable` sub-package now includes support for the `VOTable 1.3
proposed recommendation
<http://www.ivoa.net/documents/VOTable/20130315/PR-VOTable-1.3-20130315.html>`_.
Notably, this includes a new binary representation that supports masking of
any data type.

Logger
------

The Astropy logger will now no longer log exceptions by default, and will also
no longer log any warning emitted outside of Astropy. In addition, logging to
the Astropy log file (located at ``~/.astropy/config/astropy.log`` by default
on MacOS X and Linux) has also been disabled by default. This functionality is
automatically disabled for new users, but in order to see the new default
behavior, previous users of Astropy will need to edit the Astropy configuration
file (located at ``~/.astropy/config/astropy.cfg`` by default on MaxOS X and
Linux) and change the following two lines as follows::

    # Whether to log exceptions before raising them
    log_exceptions = False

    # Whether to always log messages to a log file
    log_to_file = False

Deprecation and backward-incompatible changes
---------------------------------------------

For a full-list of deprecated features and backward-incompatible changes,
please take a look at the full :doc:`../changelog`.

In Python 2.7 and above, deprecation warnings are disabled by default. If you
want to make sure you see these warnings, you can run your Python scripts with:

    $ python -Wd script.py

In addition to deprecation warnings, Astropy will also raise warnings (by
default) about changes that are not backward-compatible. These can be disabled
by doing::

    import warnings
    from astropy.utils.exceptions import AstropyBackwardsIncompatibleChangeWarning
    warnings.simplefilter('ignore', AstropyBackwardsIncompatibleChangeWarning)
