存档

‘Python’ 分类的存档

如何创建动态MOTD

2011年9月10日 没有评论

背景介绍

刚到淘宝时发现很多服务器登陆后都会显示一段AsciiArt的信息,觉得很好玩,也稍微有点用。如果你关心具体是什么,可以发送简历到jinxi.kj@taobao.com:)

如何显示静态MOTD

以上是背景,上网搜了一下,找到了相关的资料。其实很简单,只要修改了/etc/motd文件,登陆时就会被显示出来。如果需要增加颜色的话,可以参照这里的方法:“So You Like Color–The Mysterious ^[[ Characters" (http://www.linuxjournal.com/article/8603)。在vim中输入Ctrl+V后再按ESC,接下来写上颜色代码就行。有玩过term bbs的话应该会很熟悉。

如何显示动态MOTD

设置好/etc/motd后,马上就发现了一个问题,这是个静态的内容,自然就不能显示动态的信息。比如ip地址、硬盘占用、进程数等,除此之外,也没人喜欢部署多台机器时手动在motd中设置机器相关的参数,比如CPU、core数等。于是我就找到了这篇Blog: "HOWTO: Creating a Dynamic MOTD in Linux" (http://parkersamp.com/2010/10/howto-creating-a-dynamic-motd-in-linux/)。

具体方法很简单,编写一个脚本显示动态的motd,然后添加到/etc/profile中执行就行了。当然还需要禁用/etc/motd的显示。

如何禁用静态MOTD

修改: /etc/ssh/sshd_config
PrintMotd no
然后重启sshd: /etc/init.d/sshd restart or service sshd restart
更多信息参考上面提到的Blog。当然,最简单最暴力的方法是把/etc/motd文件中内容清空。

如何安装动态MOTD脚本

1. 将动态MOTD脚本放到某个目录下,比如/usr/local/bin/dynmotd。
2. 将脚本添加到/etc/profile中
修改: /etc/profile
/usr/local/bin/dynmotd

我自定义的Python脚本

由于前面我找到的Blog中使用的脚本是shell导致不方便修改,而且部分信息还是写死的;另一方面,Mac下面不能完全兼容。所以我自己用Python实现了一个类似的脚本,支持MacOS与Linux。考虑到不想把这个简单脚本搞得太复杂,所以没有用到兼容性、全面性更好地psutil。有兴趣的可以自己试试。

脚本可以到这里获取: https://github.com/chzealot/tools/tree/master/python/dynmotd

显示效果

MacOS(Darwin)下的显示效果

MacOS(Darwin)下的显示效果

Linux下的显示效果

Linux下的显示效果

单件模式:Singleton那些事

2010年3月1日 3 条评论

单件模式确保一个类只有一个实例,并提供一个全局的访问点。 –《Head First 设计模式》

有些时候,我们恰好需要这样一种设计,保证我们实例化的类在进程地址空间内只有一个实例,有了Singleton,一切似乎都简单明了,其实暗藏杀机。

1. C++实现Singleton模式简单示例

简单的C++实现示例代码

class Singleton {
    public:
        static Singleton* GetInstance(void) {
            if (Singleton::m_instance == NULL) {
                Singleton::m_instance = new Singleton();
            }
            return Singleton::m_instance;
        }
    private:
        static Singleton* m_instance;
        Singleton() {}
};

使用gtest测试,简单地测试一下每次调用GetInstance是否返回同一地址

#include 
#include 

#include 
using namespace std;
#define NEW_COUNT 4096

class Singleton {
    public:
        static Singleton* GetInstance(void) {
            if (Singleton::m_instance == NULL) {
                Singleton::m_instance = new Singleton();
            }
            return Singleton::m_instance;
        }
    private:
        static Singleton* m_instance;
        Singleton() {}
};

Singleton* Singleton::m_instance = NULL;

TEST(Singleton, sample) {
    Singleton* init_instance = Singleton::GetInstance();
    ASSERT_NE(init_instance, (Singleton*)NULL);

    Singleton* p = NULL;
    ASSERT_EQ(p, (Singleton*)NULL);

    for (int i = 0; i < NEW_COUNT; ++i) {
        p = Singleton::GetInstance();
        ASSERT_EQ(init_instance, p);
    }
}

int main(int argc, char **argv)
{
    ::testing::InitGoogleTest(&argc, argv);
    return RUN_ALL_TESTS();
}

测试结果显示没有问题:
$ ./singleton
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Singleton
[ RUN ] Singleton.sample
[ OK ] Singleton.sample (1 ms)
[----------] 1 test from Singleton (1 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (2 ms total)
[ PASSED ] 1 test.

当然,这种测试没有太大意义。给出测试结果,主要用于之后的对比。

2. python程序中调用Singleton模式的C++代码
初次接触Singleton后发现,使用Singleton模式的C++代码融合到python中有意想不到的惊喜:如果有某个类初始化需要很长时间,而之后调用的速度都很快的话,使用Singleton模式,然后在python中多次调用时,不会出现重复的初始化操作。
简单地尝试了一下

首先在singleton.cpp中添加了一行输出,标识一次初始化操作:

Singleton* Singleton::GetInstance(void) {
    if (Singleton::m_instance == NULL) {
        Singleton::m_instance = new Singleton();
        std::cout << "initialize here." << std::endl;
    }
    return Singleton::m_instance;
}

封装的代码如下
$ cat _pysingleton.cpp -n

#include 

#include "singleton.h"

using namespace std;

PyObject * GetInstance(PyObject *self, PyObject *args) {
    Singleton::GetInstance();
    return Py_BuildValue("s", NULL);
}

static PyMethodDef _pysingletonMethods[] = {
    {"GetInstance", GetInstance, METH_VARARGS, "get instance."},
    {NULL, NULL},
};

PyMODINIT_FUNC init_pysingleton(void) {
    PyObject * m;
    m = Py_InitModule("_pysingleton", _pysingletonMethods);
}

编译出来共享库_pysingleton.so
$ cat Makefile -n

CC                =       g++
FLAGS           =       -ggdb -O2 -Wall
INC               =       -I/usr/include/python2.5
LIB               =       -lgtest -lpython2.5
LLIB              =

%.o:%.cpp
        $(CC) $(FLAGS) $(INC) -c $<

_pysingleton.so: singleton.o _pysingleton.o
        $(CC) -shared $(CFLAGS) $(LLIB) $(INCLUDE) $^ -o $@

clean:
        rm -f *.so
        rm -f *.o
        rm -f *.pyc
        rm -f singleton

写个简单的python脚本测试一下在单一进程空间里,是否真的只调用一次
$ cat pysingleton.py -n

#!/usr/bin/env python
# -*- coding: utf-8 -*-

''' sample for pysingleton
'''

import os
import sys
import _pysingleton

class PySingleton:
    def __init__(self):
        _pysingleton.GetInstance()

def main():
    ''' main function
    '''
    for i in xrange(3):
        PySingleton()
    print 'Done'

if __name__ == '__main__':
    main()

结果是
$ ./pysingleton.py
initialize here.
Done

3. 多线程中使用Singleton模式
多线程里就会遇到一个double-check问题 (double-check-lock problem, DCLP)
首先,静态变量m_instance作用域是全局的,处于临界区,在多线程需要加锁。这一点很好理解。
但是,简单的在if外面加锁势必会影响性能,毕竟只有第一次调用才会生效,每次都有一次锁操作太费时,也没必要。
于是,就会有double-check方法了,伪代码为
if (Singleton::m_instance == NULL) // first check
lock
if (Singleton::m_instance == NULL) // second check

从这里可以看出,double-check有效地避免了重复锁操作问题。
然而,问题还没有结束。采用double-check的话,又会引入另一个问题。在这里
Singleton::m_instance = new Singleton();
编译器可能会做个优化,首先将一个有效的内存地址返回给m_instance,然后再调用构造函数初始化。这样很有下一次double-check中首次检查通过,但是得到的内存地址指向的是一个未经初始化的类。后果怎么样,只有天知道了。
更多细节,参考这里:http://www.ibm.com/developerworks/library/j-dcl.html

a. 最简单的修改方法是
Singleton* temp = new Singleton();
m_instance = temp;

但是一个自作聪明地编译器还是会把结果优化成难以想象的样子
b. 另一个方法就是使用volatile关键字,用以声明不受编译器优化的影响
比如 Singleton* volatile temp = new Singleton();
但是远远不够,temp是volatile,但是*temp呢,temp->foo,temp->bar...这些成员呢?所以,要改的话,得把全世界都改成volatile,除了那个lock-_-|||

4. 以上都只考虑单核,再考虑一下多核机器,问题将会更加复杂

总结:珍爱生命,远离Singleton。尤其是不要把Singleton放进通用库中,还有C++中那个叫template的高级玩意儿,都搞在一起老天都很难保证会发生些什么。


1. 这里也在讨论单例模式Why Singletons Are Controversial
2. 代码托管到Google Code,欢迎围观、讨论、批评、挑bug:点此进入(http://code.google.com/p/zldemo/source/browse/#svn/trunk/blog/singleton_everything)
(墙内用户请不要使用https访问)

Python罕见秘笈搜集

2008年10月13日 1 条评论

features of Python

From: http://stackoverflow.com/questions/101268/hidden-features-of-python

  • Chaining comparison operators:

    >>> x = 5

    >>> 1 < x < 10

    True

    >>> x < 10 < x*10 < 100

    True

    >>> 10 > x <= 9

    True

    >>> 5 == x > 4

    True

    In case you’re thinking it’s doing, ’1 < x’, which comes out as True, and then comparing ‘True < 10′, which is also True, then no, that’s really not what happens (see the last example.) It’s really translating into ’1 < x and x < 10′, and ‘x < 10 and 10 < x * 10 and x*10 < 100′, but with less typing and each term is only evaluated once.

  • Readable regular expressions

    In Python you can split a regular expression over multiple lines, name your matches and insert comments.

    Example verbose syntax (from Dive into Python):

    >>> pattern = “”"

        ^                   # beginning of string

        M{0,4}              # thousands – 0 to 4 M’s

        (CM|CD|D?C{0,3})    # hundreds – 900 (CM), 400 (CD), 0-300 (0 to 3 C’s),

                            #            or 500-800 (D, followed by 0 to 3 C’s)

        (XC|XL|L?X{0,3})    # tens – 90 (XC), 40 (XL), 0-30 (0 to 3 X’s),

                            #        or 50-80 (L, followed by 0 to 3 X’s)

        (IX|IV|V?I{0,3})    # ones – 9 (IX), 4 (IV), 0-3 (0 to 3 I’s),

                            #        or 5-8 (V, followed by 0 to 3 I’s)

        $                   # end of string

        “”"

    >>> re.search(pattern, ‘M’, re.VERBOSE)

    Example naming matches (from Regular Expression HOWTO)

    >>> p = re.compile(r‘(?P<word>\b\w+\b)’)

    >>> m = p.search( ‘(((( Lots of punctuation )))’ )

    >>> m.group(‘word’)

    ‘Lots’

  • Decorators

    Decorators allow to wrap a function or method in another function that can add functionality, modify arguments or results, etc. You write decorators one line above the function definition, beginning with an “at” sign (@).

    Example shows a print_args decorator that prints function’s arguments before calling it:

    >>> def print_args(function):

    >>>     def wrapper(*args, **kwargs):

    >>>         print
    ‘Arguments:’, args, kwargs

    >>>         return
    function(*args, **kwargs)

    >>>     return wrapper

     
     

    >>> @print_args

    >>> def write(text):

    >>>     print text

     
     

    >>> write(‘foo’)

    Arguments: (‘foo’,) {}

    foo

  • iter() can take a callable argument

    For instance:

    def seek_next_line(f):

        for c in iter(lambda: f.read(1),‘\n’):

            pass

    The iter(callable, until_value) calls repetitively the callable and yields its result until the callable returns _until_value_.

  • Creating generators objects

    If you write

    x=(n for n in foo if bar(n))

    you can get out the generator and assign it to x. Now it means you can do

    for n in x:

    The advantage of this is that you don’t need intermediate storage, which you would need if you did

    x = [n for n in foo if bar(n)]

    In some cases this can lead to significant speed up.

  • Futures and the “With” Statement

    There’s a special module in Python called __future__. Some new language features end up in this module for testing, and to use them you have to explicitly import them from here. One such feature which is a favorite of mine is the “with” statement, which is currently present in __future__ in version 2.5, but will be part of the language in the soon-to-be-released 2.6 and 3.0 versions.

    The reason it is in __future__ is because it makes both with and as keywords, which could break existing code.

    I have used the “with” statement in 2.5 a lot because I think it’s a very useful construct, here is a quick demo:

    from __future__ import with_statement

     
     

    with open(‘foo.txt’, ‘w’) as f:

        f.write(‘hello!’)

    What’s happening here behind the scenes, is that the “with” statement calls the special __enter__ and__exit__ methods on the file object. Exception details are also passed to __exit__ if any exception was raised from the with statement body, allowing for exception handling to happen there.

    What this does for you in this particular case is that it guarantees that the file is closed when execution falls out of scope of the “with” statement‘s body, regardless if that occurs naturally or whether an exception was thrown. It is basically a way of abstracting away common error-handling code.

    Other common use cases for this include locking with threads and database transactions.

    For more information on how to use this and how to implement your own “with” statement compatible objects read PEP 343

  • The step argument in slice operators. For example:

    a = [1,2,3,4,5]

    >>> a[::2]  # iterate over the whole list in 2-increments

    [1,3,5]

    The special case x[::-1] is a useful idiom for ‘x reversed’.

    >>> a[::-1]

    [5,4,3,2,1]

  • From 2.5 onwards dicts have a special method __missing__ that is invoked for missing items:

    >>> class
    MyDict(dict):

    …  def __missing__(self, key):

    …   self[key] = rv = []

    …   return rv


    >>> m = MyDict()

    >>> m["foo"].append(1)

    >>> m["foo"].append(2)

    >>> dict(m)

    {‘foo’: [1, 2]}

    There is also a dict subclass in collections called defaultdict that does pretty much the same but calls a function without arguments for not existing items:

    >>> from collections import defaultdict

    >>> m = defaultdict(list)

    >>> m["foo"].append(1)

    >>> m["foo"].append(2)

    >>> dict(m)

    {‘foo’: [1, 2]}

    I recommend converting such dicts to regular dicts before passing them to functions that don’t expect such subclasses. A lot of code uses d[a_key] and catches KeyErrors to check if an item exists which would add a new item to the dict.

  • Doctest: documentation and unit-testing at the same time.

    Example extracted fom python documentation:

    def factorial(n):

        “”"Return the factorial of n, an exact integer >= 0.

     
     

        If the result is small enough to fit in an int, return an int.

        Else return a long.

     
     

        >>> [factorial(n) for n in range(6)]

        [1, 1, 2, 6, 24, 120]

        >>> factorial(-1)

        Traceback (most recent call last):

            …

        ValueError: n must be >= 0

     
     

        Factorials of floats are OK, but the float must be an exact integer:

        “”"

     
     

        import math

        if
    not n >= 0:

            raise
    ValueError(“n must be >= 0″)

        if math.floor(n) != n:

            raise
    ValueError(“n must be exact integer”)

        if n+1 == n:  # catch a value like 1e300

            raise
    OverflowError(“n too large”)

        result = 1

        factor = 2

        while factor <= n:

            result *= factor

            factor += 1

        return result

     
     

    def _test():

        import doctest

        doctest.testmod()    

     
     

    if __name__ == “__main__”:

        _test()

  • Descriptors

    They’re the magic behind a whole bunch of core Python features.

    When you use dotted access to look up a member (eg, x.y), Python first looks for the member in the instance dictionary. If it’s not found, it looks for it in the class dictionary. If it finds it in the class dictionary, and the object implements the descriptor protocol, instead of just returning it, Python executes it. A descriptor is any class that implements the getset, or del methods.

    Here’s how you’d implement your own (read-only) version of property using descriptors:

    class
    Property(object):

        def __init__(self, fget):

            self.fget = fget

     
     

        def __get__(self, obj, type):

            if obj is
    None:

                return
    self

            return
    self.fget(obj)

    and you’d use it just like the built-in property():

    class
    MyClass(object):

        @Property

        def foo(self):

            return
    “Foo!”

    Descriptors are used in Python to implement properties, bound methods, static methods, class methods and slots, amongst other things. Understanding them makes it easy to see why a lot of things that previously looked like Python ‘quirks’ are the way they are.

    Raymond Hettinger has an excellent tutorial that does a much better job of describing them than I do.

  • Sending values into generator functions. For example having this function:

    def mygen():

      “”"Yield 5 until something else is passed back via send()”"”

      a = 5

      while
    True:

        f = yield(a) #yield a and possibly get f in return

        if f: a = f  #store the new value

    You can:

    >>> g = mygen()

    >>> g.next()

    5

    >>> g.next()

    5

    >>> g.send(7)  #we send this back to the generator

    7

    >>> g.next() #now it will yield 7 until we send something else

    7

  • One line Variable value swapping

    >>> a = 10

    >>> b = 5

    >>> a, b = b, a

     
     

    >>> print a

    5

    >>> print b

    10

    a will have the value of b and so on.

    This is a side effect of python packing and unpacking feature.

  • enumerate

    Wrap an iterable with enumerate and it will yield the item along with it’s index.

    For example:

    >>> a = ['a', 'b', 'c', 'd', 'e']

    >>> for index, item in enumerate(a): print index, item


    0 a

    1 b

    2 c

    3 d

    4 e

    >>>

  • Creating new types at runtime

    >>> NewType = type(“NewType”, (object,), {“x”: “hello”})

    >>> n = NewType()

    >>> n.x

    “hello”

    which is exactly the same as

    >>> class
    NewType(object):

    >>>     x = “hello”

    >>> n = NewType()

    >>> n.x

    “hello”

    Probably not the most useful thing, but nice to know.

    Edit: Fixed name of new type, should be NewType to be the exact same thing as with class statement.

     
     

    查看更多:http://stackoverflow.com/questions/101268/hidden-features-of-python

    这个页面里收集了很多python技巧

    Onenote不错,再试一下Onenote发帖,呵呵

     
     

分类: linux, Python 标签: , ,

贴个python代码测试下coolcode插件

2008年5月25日 没有评论

python utils
平时写的几个python util,包括了读写文件和http下载;另外的dprint用于调试的,没办法,python print中文list时没办法直接显示

# -*- coding: gbk -*-
#!/usr/bin/env python2.4

import os
import sys
import urllib
import urllib2
import cookielib
from types import *
from urllib import urlencode
def LoadFile(filename, mode=’r'):
“”"安全读取文件内容
1.安全读取文件,并返回文件内容.
2.open 函数可能引发 IOError 异常。(可能是文件不存在。)
3.read 方法可能引发 IOError 异常。(可能磁盘有坏扇区,或它在一个网络驱动器上,而网络刚好断了。)
4.一旦文件通过 open 函数被成功地打开,我们应该绝对保证把它关闭,即使是在 seek 或 read 方法引发了一个异常时。try…finally 块可以用来:在 finally 块中的代码将总是 被执行,甚至某些东西在 try 块中引发一个异常也会执行。可以这样考虑,不管在路上发生什么,代码都会被 “即将灭亡” 地执行。
5.最后,处理我们的 IOError 异常。它可能是由调用 open、seek 或 read 引发的 IOError 异常。这里,我们其实不用关心,因为将要做的事就是静静地忽略它然后继续。(记住,pass 是一条不做任何事的 Python 语句。) 这样完全合法,“处理” 一个异常可以明确表示不做任何事。它仍然被认为处理过了,并且处理将正常继续,从 try…except 块的下一行代码开始。
“”"
content = ”
try:
fp = open(filename, mode)
try:
content = fp.read()
finally:
fp.close()
except IOError, e:
print ‘#Error:’, ‘open ‘+filename, str(e)
return content
def WriteFile(filename, content, mode=’w'):
“”" 安全写入文件,对异常进行恰当的处理 “”"
try:
fp = open(filename, mode)
try:
fp.write(content)
finally:
fp.close()
except IOError, e:
print ‘#Error:’, ‘open ‘+filename, str(e)
def HttpDownload(url):
“”"下载http页面文件
安全下载http页面文件,增加异常判断和处理
“”"
content = ”
# req = urllib2.Request(url)
try:
fp = urllib2.urlopen(url)
try:
content = fp.read()
finally:
fp.close()
except Exception, e:
print ‘#Error: open’,url,’failed!’,str(e)
return content
def dprint(content):
“”"输出变量内容
针对list tuple dict的元素单独使用print,用于调试时查看非Ascii内容
“”"
conType = type(content)
if conType in [TupleType,ListType]:
for i in content:
dprint(i)
elif conType is DictType:
for name,value in content.items():
if name in [TupleType,ListType,DictType]:
dprint(name)
else:
print name+’:',
dprint(value)
else:
print content
def Login(url, userinfo):
cj = cookielib.LWPCookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
opener.addheaders = [\
("User-agent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031107 Debian/1.5-3"),\
("Accept", "text/html, image/jpeg, image/png, text/*, image/*, */*")]
postdata = urlencode(userinfo)
r = opener.open(url,urlencode(userinfo))

if __name__==’__main__’:
print ‘Test Util Start:’
a = ['Losthit','中关村',{'zealot':'北京','test':'海淀区'}]
dprint(a)
print HttpDownload(‘http://www.baidu.com’)

分类: Python 标签: , ,