back to article First Python feature release under new governance model is here, complete with walrus operator (:=)

Python's new feature release – version 3.8 – has landed, the first since June 2018. Python is the third most popular language in the Redmonk language rankings and fourth in the StackOverflow developer survey – second if you do not count HTML/CSS and SQL, behind JavaScript but ahead of Java. The download page identifies 17 " …

  1. hellwig

    What was wrong with C's implementation?

    Sure, using '=' in a conditional could lead to mistakes, but "if ((x = SomeFunc()) > 0) { /* use x */ }" seemed to work fine for C, where did Python diverge?

    1. Tom 38 Silver badge

      Re: What was wrong with C's implementation?

      This, and many other queries and alternate suggestions, are dealt with in the PEP:

      https://www.python.org/dev/peps/pep-0572/#why-not-just-turn-existing-assignment-into-an-expression

      PEPs are often quite interesting to read.

      1. FöoBär

        Re: What was wrong with C's implementation?

        I think the julia's approach is just right. = is an expression as in C, but its use directly in an if condition statement, is forbidden.

        if n = 0 # -> error

        # do something

        end

        if (count = read_file()) > 0 # -> ok

        # do something with count

        end

    2. Anonymous Coward
      Anonymous Coward

      @hellwig - Re: What was wrong with C's implementation?

      Nothing wrong. Only thing is Python insists to do the same thing as C in a more complicated manner but in the name of simplicity :)

      OK, OK, just let me get my coat.

    3. Anonymous Coward
      Anonymous Coward

      Re: What was wrong with C's implementation?

      It's not just that it "could" lead to mistakes, it's just that it makes them practically inevitable, especially when you consider the C idiom of writing something like "if (x = SomeFunc()) ..." for a test of non-zero return. In fact, some variation of "if (x = 0) ..." might be the single commonest error in C code.

      In fairness, this occurs not just because of the use of '=' for assignment, but because of the intersection of three design choices in C: (1) the assignment operator; (2) that C is an expression-based language, so the assignment "x = 0" is also a valid expression; and (3) the lack of a formal Boolean type, instead having the convention of 0 as False, everything else is True. Change any one of those and the most pernicious version of the trap goes away -- although the example you provided still exists.

    4. martinusher Silver badge

      Re: What was wrong with C's implementation?

      >but "if ((x = SomeFunc()) > 0) { /* use x */ }"

      This works fine but I tend to avoid such constructs because people who try to cut down keystrokes by using such forms often are tempted to leave out additional -- and what they regard as redundant -- parentheses. Most compilers will generate the same code if the assignment is given its own line so I'd only use this type of construct if 'x' was something like a register variable that I was using to second guess the compiler's optimizer.

    5. Bill Gray

      Re: What was wrong with C's implementation?

      I'd be torn about this. (Though I don't really have a dog in the fight. I've programmed in C daily for decades, but do virtually nothing in Python.)

      Most C and C++ compilers (that I've tried, at any rate) will give you a warning in almost all cases where you use = in a place where == ought to have been used. But you have to have compiler warnings turned on (I'm _amazed_ at how many people don't do this, but... they don't.)

      Much of my code is open source, and may be read/used by people with a range of coding skills and C knowledge. I avoid assignments within conditionals in any code that other people will see, and rarely use it even in code that will never leave my machine. It's not that an assignment within a conditional is always terrible, but it _usually_ indicates that the bit of code in question should be rewritten and made clearer. Certainly in the instance you give, where I would argue that the following is easier to read:

      x = SomeFunc( );

      if( x > 0) { /* use x */}

      1. Graham 32

        Re: What was wrong with C's implementation?

        > it _usually_ indicates that the bit of code in question should be rewritten and made clearer

        Very strongly agree. Good code is written once and read many times. Making code quicker to write* is often not a time-saver in the long term.

        * as in less keystrokes. Constructs/libraries/etc that let you reduce the complexity of code is good.

        1. Bill Gray

          Re: What was wrong with C's implementation?

          "...Good code is written once and read many times..."

          That took me a while to figure out. I have quite a bit of code that I wrote starting in 1992, and I wasn't really thinking about the possibility that I might be looking at it 27 years later, wondering what the %&!# my younger self was trying to do. Eventually, I came around to this point of view (written concerning a clever, but impenetrable algorithm):

          https://github.com/Bill-Gray/lunar/blob/master/date.cpp#L103

          /* Personally, I like to be able to look at a chunk of code and see what it means. It should resemble the Reformation view of the Bible: anyone can read it and witness the truth thereof. */

      2. ThomH

        Re: What was wrong with C's implementation?

        C++17 does the best stab at this yet with its if-statement initialisers, I think:

        if(int x = someFunc(); x > 0) { /* use x */ }

        So, much the same as the way that 'for' has always worked. Note that x is scoped so that it isn't visible outside the if. Unlike if you had assigned x prior to evaluating the if. If you didn't want that then you could declare the x first and assign it only within the if, but that'd be a bit silly, even if still mildly less error-prone than an old if(x = ...).

      3. Brewster's Angle Grinder Silver badge

        if (auto x = SomeFunc(); x > 0 ) { /*...*/ }

      4. swm Silver badge

        Re: What was wrong with C's implementation?

        It is more useful in a while statement:

        while(data = strm.get()) != EOF) {

        process data

        }

        although C++

        while(strm.get(data)) {

        process data

        }

        also works.

        1. Richard Plinston Silver badge

          Re: What was wrong with C's implementation?

          > It is more useful in a while statement:

          > while(data = strm.get()) != EOF) {

          Python doesn't need that assignment, nor test for EOF:

          fd = open(filename)

          for data in fd:

          process data

          Nor does it suffer from buffer overflow.

  2. Alan J. Wylie

    :=

    Oooh. Just like Algol 60's assignment operator.

    1. Anonymous Coward
      Anonymous Coward

      @Alan J. Wylie - Re: :=

      Yep! History is repeating itself, soon we'll get some Cobol incorporated into Python. Just wait until cool kids will discover it.

    2. LDS Silver badge

      Re: :=

      But made well known to the masses by Pascal (which borrowed it from ALGOL).

      1. Anonymous Coward
        Anonymous Coward

        Re: :=

        Let's not forget Smalltalk!!!

        Still helping to keep a roof over my head.

        Posting as AC - as I am working for an Electricity company

        1. swm Silver badge

          Re: :=

          The original Smalltalk used a left arrow for assignment. This was before Smalltalk-80.

    3. Richard Plinston Silver badge

      Re: :=

      > Algol 60's assignment operator.

      No. In C the equal sign (=) is an _operator_ that causes assignment in an expression. 'x = 0' is an expression. This is why assignment can be done in an if statement evaluation.

      In Algol and Pascal the ":=" is part of the syntax of an assignment statement. 'x := 0' is a statement, not an expression, because ':=' is not an expression operator.

      This is a fundamental difference between Pascal like languages and C like. Assignment in an 'if' can lead to errors, not only because '=' can be mistakenly written instead of '==' but because of 'short circuiting' evaluations may fail to do the intended assignment.

  3. GrahamRJ

    Why TF would anyone want to *add* this? It's a well-recognised anti-pattern in C and C++, to the extent that coding standards such as MISRA explicitly ban its use. At least the "walrus operator" is relatively harder to use accidentally, which is its main reason for being excluded from C/C++ coding standards; but in practise it really doesn't bring you anything except less-clear code.

    1. Tom 38 Silver badge

      Look how it can make existing code clearer.

      1. Phil O'Sophical Silver badge

        Nothing that couldn't be achieved by well-known syntax like parentheses, without the confusion of a new operator. It just creates different ways to do the same thing, depending on context. Horrible.

        1. Anonymous Coward
          Anonymous Coward

          Yes, and...

          Not only is this construct an anti-pattern in C/C++, creating different ways to do the same thing is the most fundamental anti-pattern in Python.

          Sometimes I feel that a lot of people have forgotten why some of the worst elements of C are in the language, namely to allow the programmer to express optimizations that a compiler from the early 1970s might miss, but which modern compilers / interpreters have been capable of implementing for decades. There's no reason to perpetuate them in modern languages.

          1. Anonymous Coward
            Anonymous Coward

            Re: Yes, and...

            CPython is hardly know for its sophisticated optimisations (it's like a toy interpreter from CS a class that grew too big).

            1. Charlie Clark Silver badge

              Re: Yes, and...

              The sort algorithm, which has since become a reference model, would suggest you don't know what you're talking about.

      2. Spudley

        Yes, the original code is ugly, but there will be be better ways to tidy up it up than using this operator.

        Fundamentally, the problem with assignment within an expression is that it is a one-line violation of CQS (command / query separation), which frankly is one of the most important rules you can learn for writing reliable, good quality code.

        1. tekHedd

          QCS

          aka "one line of code should do one thing", one of the most fundamental rules of good style (and also making debugging less of a PITA). But nobody follows that rule; if you do all the "experienced" programmers will make fun of you.

      3. Anonymous Coward
        Anonymous Coward

        How many ways are there for the "else if" statement to be constructed in different programming languages?

        else If

        elsif

        elseif

        elif

        ...

        1. Alan J. Wylie

          if ... then ... else

          bash $ true && echo yes || echo no

          yes

          bash $ false && echo yes || echo no

          no

          $

          $ perl -e '1 == 1 and print "yes" or print "no";'

          yes$

          $ perl -e '1 == 2 and print "yes" or print "no";'

          no$

          (C)

          result = a > b ? x : y;

      4. GrahamRJ

        Except it doesn't. For me, the "improved" version is much harder to follow at a quick scan. One of the key features for Python is that it *didn't* have bizarrely arcane language grammar, so it was very suitable as a training language. And now they've added some. Yay. Not. If that's the best example they can find, I'd say that my most charitable assessment would be "no proven benefit, some proven negative".

        If it was already in the language (like C/C++) then fine, leave it be. When it isn't already in the language, it's a ton of extra work for no reason other than one person (GvR) thinks it'd be neat. And that's certainly a compelling reason for GvR to step down, if he's got bored with the Python language in the form which made it successful and decided to drop in changes which break its reason for being successful in the first place.

      5. rvt

        None of that code is really clear, of almost none of the variables I can see what type they are and how they are evaluated.

      6. Charlie Clark Silver badge

        In previous discussions that were similar Guido tended to follow the argument that to do nothing would cause the least problem. This is exactly the kind of the syntax that, because it's occasionally useful, it gets used all over the place where not only it isn't useful but in fact downright confusing, aka dangerous.

        Seeing as it's backwards incompatible

      7. Anonymous Coward
        Anonymous Coward

        re: Look how it can make existing code clearer.

        The first example is clearer but in part because it "elseifs" rather than nesting. The other examples are just shorter, not clearer at all.

    2. Spudley

      I agree.

      There are no circumstances that make this a good practice, regardless of whether you have a distinct operator for it. The only reason for wanting to do it is because of some misguided belief that fewer lines of code makes your code neater. It does not.

      1. Anonymous Coward
        Anonymous Coward

        i'm no advocate for this operator. I haven't even used python for over a year, but I think you are mistaken in your belief that it is just there for fewer lines of code and therefore makes your code look neater.

        In most programming languages you could remove 20%+ of the spec and still be able to achieve what you want to do another way. However simplification of certain functions or multiple error and sanity checks can be avoided making the code much easier to write, parse, debug and less prone to unintended bugs and security risks. To a casual observer this could be interpreted as just reducing lines and making it neater.

        I can think of a number of areas that this operator could be used and I would expect this to and fro about the usefulness has happened regularly over the last few months of trying to get this sanctioned. However people intimately involved in the construction of this language are convinced it is useful, but you may be more qualified in your statement that "There are circumstances that make this good practice" (maybe you missed out a IMHO tag?)

        For everyone else, either use it or don't it is up to you. If you have to parse some code that uses it even though you choose not to, then either appreciate the ease of parsing it or decide that your eyes would rather be scratched out than have to try to acknowledge it and skip over any such functions that use it.

        It took me quite a while to parse the lambda (=>) and namespace alias qualifier (::) in C# when I started trying to debug it.

        1. GrahamRJ

          It is explicitly banned as an anti-pattern by MISRA, who are one of the leading organisations devoted to setting software development standards. The reason is twofold: combining assignment and test in one line makes it harder to follow (as assessed by MISRA, who set the coding standard based on code being easily reviewable and maintainable by other people); and it also makes it impossible to tell whether the assignment was *genuinely* supposed to be there or whether the coder simply made a typo.

          So not just IMHO, but also in the less-humble opinion of all people involved in setting a leading safety-related software development standard. Incidentally, I've also met the same rule in in-house C coding standards for automotive and aerospace software, again set by the senior engineers of those organisations.

          I'd be genuinely interested to find people who can advocate for it with any argument other than "it's less lines of code", an argument which we all know (or should know) is fallacious.

          1. Anonymous Coward
            Anonymous Coward

            MISRA aren't really relevant when talking about Python or these use cases, I would suggest. Not really a safety critical embedded system or a vehicle system. I mean even a = and == for assignment and test is an easy to miss potential security flaw.

            On a higher level, the Java string comparator vs a normal == comparator has caught me out a few times. Easy to introduce bugs with that one.

            1. GrahamRJ

              That's kind of missing the point of the coding standard. It doesn't matter whether the code is going into Flappy Birds or a nuclear power station. The point is that applying the coding standard gives you code which is less likely to be wrong in the first place, and thereafter can be easily reviewed and understood by most people at a glance.

              Many languages have known holes which users will frequently fall into. A coding standard is about covering over those holes. Regardless of whether the bottom of the hole just contains dogshit or a claymore mine, not putting your foot in the hole in the first place is still a good thing.

              So when the architect of Python *intentionally* creates a new hole, which we have been fully aware of for the last 30 years or more (the original Lint warned about this) and furthermore was intentionally absent from the language in the first place, and he says it's for the alleged benefit of novice coders who are unable to formulate an if-else statement, I can only be sceptical about his competence not just as an architect but also as a software engineer.

              It's not uncommon that people get promoted out of day-to-day technical work, and that's fine. But then accepting that you're no longer as technically adept as you used to be is essential. I'm glad he's stepped down; it's just a shame he's tracked dogshit through the office before he left.

          2. disgustedoftunbridgewells Silver badge

            I find it easier to follow as long as it's kept simple.

            The alternatives are either nesting which makes it harder to track, or assigning everything before an if/elseif block which in some situations is wasteful if the values are assigned the return value of computationally expensive functions.

      2. The Indomitable Gall

        If you look at the PEP, one of Guido's justifications was that reviewing existing Python code, he was finding plenty of examples of people duplicating work (eg [ f(x) for x in x_list if f(x)>0 ]) or doing redundant work to avoid nested ifs, and he saw this as a solution for real-world problems.

        The other advantage is removing what the linguist part of me would call "long-range dependencies". The closer the assignment is to its use, the easier it is to reason through the code. Particularly, mathematicians and scientists are used to reasoning through things in semantically dense formulas, and less used to the step-by-step imperative style.

        1. Charlie Clark Silver badge

          I'm not sure that the syntax change will stop people writing code to avoid nested if statements.

          DSLs are probably the way to go for the small group of people (relatively) who need this.

          Python has a history of introducing syntactic changes only to reverse them later (backticks, print >>, map, reduce, etc.). Some stuff stays because it is genuinely useful but this can take years of refinement before it becomes standard. My initial reaction to this change is that it is, unfortunately, yet more special use syntax forced on the rest of us. It certainly won't aid readability and the improvements are marginal. But let's see what the take up is in the next few years.

    3. The Indomitable Gall

      When you remove the possibility of =/== substitution bugs, the main danger of C-style assignment is gone.

      The main reason I see the walrus as a good thing is in list comprehensions.

      Consider :

      newlist = [ f(x) for x in oldlist if f(x) > 0 ]

      Now imagine the that f(x) is O(^n) or O(n!) -- you've either got to take the performance hit of running it twice or expand the code out. For a scientist or mathematician (and academia is a major target audience for Python), this one line, pseudo-mathematical, pseudo-functional approach is much more readable.

      My personal preference would have been for local variable declarations, eg:

      newlist = [ result for x in oldlist if result > 0 where result = f(x) ]

      but that's not the way they chose to go. Instead we have:

      newlist = [ result for x in oldlist if (result:= f(x)) > 0 ]

      ...which is less restrictive than my way, and Guido set out good reasons for it. But when you look back at the examples, I think it boils down to this: a great many users of Python aren't immersed in the imperative programming style the way most pro software devs are, and the reliance on branching blocks and lines is more confusing to them as it renders code logic implicit. I'm surprised it took Guido as long as it did (v2.4) to introduce conditional expressions, actually, as that's really useful for bringing a single mathematical function into a single line.

      1. tekHedd

        newlist = [ result for x in oldlist if (result:= f(x)) > 0 ]

        I get it. Since it's currently considered pythonic to code in nearly unreadable one-liners, the additional gotchas are largely moot.

        1. Frank Marsh
          Thumb Up

          Re: newlist = [ result for x in oldlist if (result:= f(x)) > 0 ]

          Thank you. Literally laughed out loud. I'm still chuckling.

    4. thames

      It will be a very useful feature in list comprehensions, where it offers the potential for performance improvements as you no longer have to repeat an operation in order to use the result several times in the same expression.

      Here's an example directly from PEP 572.

      results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0]

      Without it you would have to write

      results = [(x, f(x), x/f(x)) for x in input_data if f(x) > 0]

      The other alternative would be to unroll the comprehension into a for loop, but comprehensions (of various types) are generally faster as they have less loop overhead. This means they are a common optimisation method, and anything which makes them even faster is generally a good thing.

      I run into this type of problem on a regular basis and look forward to using it once version 3.8 percolates out into the common Linux distros.

      1. Charlie Clark Silver badge

        This means they are a common optimisation method, and anything which makes them even faster is generally a good thing.

        Premature optimisation is the root of all evil and a great way to waste time. I always advise people to write clear code and profile if they need more speed. Comprehensions don't necessarily loop much faster, not that Python's loops are that slow, but they do allocate less memory. That said, running nested loops through a JIT like PyPY or Cython will generally lead to C-style speeds.

    5. martinusher Silver badge

      >It's a well-recognised anti-pattern in C and C++

      Its an assignment operator in ALGOL -- the great-grandparent of all block structured languages. It differs from an equate because its an assignment to a variable reference rather than an identity -- that is, it has a rather nice property of being unambiguous, you don't need to know the context of its use to understand its meaning.

      So, with regard to "less clear code", it actually does the opposite -- its utterly unambiguous. But then people have got so used to using '=' as assignment, as identity or as a comparison that they probably don't even realize there is a difference.

      1. Richard Plinston Silver badge

        > Its an assignment operator in ALGOL

        No. Assignment is a statement in Algol (and Pascal), not an operator.

        > the great-grandparent of all block structured languages

        No. Algol is the ancestor of Pascal like languages. C derives from CPL via BCPL and B, and not from Algol.

    6. chololennon

      At least in C++ is not always an anti-pattern. What is more, the pattern to use the dynamic_cast operator with pointers is the following:

      if (Foo* foo = dynamic_cast<Foo>(some_pointer)) {

      // foo usage

      }

      1. Brangdon

        re: dynamic_cast

        That dynamic_cast example doesn't have an assignment. It declares and initialises a new variable instead. It's much safer.

  4. Notas Badoff

    Bliss and hisses

    So we're going to end up converging on the features of Bliss decades later? Maybe not, but the feature by feature intertwining of languages is getting more than a little confusing. Do we do exceptions here like there or like that other language? Wait, what language am I staring at?

    Also, I guess the article said GvR was a proponent of the expression assignment operator. Could the opposition have been described as reactionary, wanting to keep the language pure from borrowed features? I think another major religion is having this problem right now. I think the BDFL there is named Francis?

    1. Jason Bloomberg Silver badge

      Re: Bliss and hisses

      I imagine part of the debate raged around 'common sense' and what programmers might like versus 'Python Purity'. For example "a = 1" being okay but not "a := 1", "a = b = 2" being okay but not "a := b := 2" nor "a = b := 2", though "a = ( b := 2 )" is acceptable but frowned upon.

      I would have just defined "=" for both purposes, or had "=" and ":=" as synonyms so people can use whatever they choose to.

  5. Anonymous Coward
    Anonymous Coward

    eggman

    No ! Everyone knows that a walrus needs his lost bukkit

    1. Ken Shabby Bronze badge
      Linux

      Re: eggman

      Seems he blew a seal

  6. This post has been deleted by a moderator

    1. Anonymous Coward
      Anonymous Coward

      Re: Death to all programming languages!

      Add enough AI, and humans, themselves, are no longer necessary and become unwelcome constraints on the functioning of the AI.

      1. Il'Geller

        Re: Death to all programming languages!

        Programming language performs two functions:

        - retrieval,

        - and processing, calculating of the found.

        AI is very good for the retrieval and almost not suitable for calculation.

    2. eldakka Silver badge

      Re: Death to all programming languages!

      If you have no programming languages, how do you create an AI in the first place?

      1. Il'Geller

        Re: Death to all programming languages!

        You got me!

        Actually I don't know how it's going to be... Let's see? What they will come up with?

    3. Psmo Bronze badge

      Re: Death to all programming languages!

      Sure, but if you can't 'prove' that a system has deterministic behaviour you're going to have trouble passing existing test suites.

      Sounds expensive.

      And how would you debug a system like that?

      1. Il'Geller

        Re: Death to all programming languages!

        Now a human operator/ programmer decides whether an "operator walrus" is required, based on a spec. In future AI will structure the same spec automatically and convert it into an action, adding the "operator walrus" by itself, as synonymous clusters. That is, there will be a fix in a dialog mode, without the participation of the programmer.

        Money is saved on the programmer and the license fees for the soft and language.

        1. Robert Grant

          Re: Death to all programming languages!

          Ah, "in the future". The promise of replacing programmers has been a cash cow since the 1970s.

  7. JacobZ

    Irony

    I just wanted to note the irony that the entire discussion so far is about the specific feature, and barely at all about the new governance process...

    1. Spudley

      Re: Irony

      I guess that in itself is a good commentary on why the governance process became necessary.

  8. druck Silver badge
    Unhappy

    But what about the GIL?

    Nothing in 3.8 stands out as a must have, or even more than mildly useful. Where progress is desperately needed is in threading, which sucks on Python due to the Global Interpreter Lock. Yes there is multiprocessing, which is just about usable on Linux, but completely hopeless on forkless Windows.

    1. Psmo Bronze badge

      Re: But what about the GIL?

      Not all Python interpreters have a GIL.

      CPython is the reference implementation, so you often have to do without the latest features or port them yourself (usually not hard).

      Stackless, or PyPy for example can provide enormous gains on web server work and massive-multithreading.

      C extensions are trickier, though usually just a header-file-swap away.

      1. druck Silver badge

        Re: But what about the GIL?

        I've found PyPy to be far worse for multi-threading and multi-processing on both Linux and Windows.

        1. Psmo Bronze badge

          Re: But what about the GIL?

          There isn't enough information for your comment to be useful.

          Compared to what? CPython? Do you have benchmarks?

          Edit: looking over the link above, it seems that for multi-microthreaded applications (like a raytrace or game application) PyPy is indeed worse. For tiny threads, I imagine the JIT compiler is giving a performance hit.

          I also see that some frameworks like Django performs worse with PyPy.

    2. LDS Silver badge

      "which is just about usable on Linux, but completely hopeless on forkless Windows."

      Not so hopeless - "spawning" a process is slower and the semantic is different from forking, but it does work and if the processes are long-lived it's not usually an issue. " Note that safely forking a multithreaded process is problematic." (from https://docs.python.org/3/library/multiprocessing.html).

      BTW, in 3.8 "spawn" became the default method is macOS too.

      But I agree the GIL in CPython is something they have to work on.

    3. fajensen Silver badge
      Angel

      Re: But what about the GIL?

      I think threads are generally Evil. Except in Erlang, which was designed from the start to handle threads correctly and efficiently. If not Erlang, Processes is the way to go, at least when one values ones sanity and cares about not getting support calls.

      Threads too often are too low-level and kinda glommed-on 'because X has it' to be conveniently managed and instrumented and yet they are, in most cases, especially Java, still hideously bloated to obscene levels of resource usage, levels where one might just use a damn process and be done with it faster and easier!

      FreeRTOS, being an embedded therefore 'has-to-work'-OS, uses full processes and a mailbox system. In FreeRTOS a process is a few hundred bytes, so it really doesn't make sense to use threads.

    4. TheGreatCabbage

      Re: But what about the GIL?

      Python 3.8 adds a really nice shared memory library for multiprocessing.

      I'm developing a cross platform program with multiprocessing and it isn't too bad on Windows.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019